Re: [Twisted-Python] Incremental vs. Black

[Moshe Zadka]

On Mon, Aug 10, 2020, at 07:29, Barry Scott wrote:
> On Saturday, 1 August 2020 18:14:15 BST Moshe Zadka wrote:
> > Hi all,
> > 
> > I opened https://github.com/twisted/incremental/issues/55 a few days ago, 
> > and I'm wondering if anyone has any objection to me making a fix. I'm using 
> > incremental at work on a project that uses black, so this just means I have 
> > to run black each time there is a version update -- not the end of the 
> > world, but annoying nonetheless.
> 
> Surely its not appropriate to run black on third party code?

Because `_version.py` is designed to be checked in to your code, is reviewed in 
a PR along with other code, etc.

Moshe Z.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Incremental vs. Black

[Moshe Zadka]

Hi all,

I opened https://github.com/twisted/incremental/issues/55 a few days ago, and 
I'm wondering if anyone has any objection to me making a fix. I'm using 
incremental at work on a project that uses black, so this just means I have to 
run black each time there is a version update -- not the end of the world, but 
annoying nonetheless.

Moshe Z.___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] pydoctor 20.7.0

[Moshe Zadka]

Indeed, let me join on to the thanks! Awesome work Maarten!

On Fri, Jul 3, 2020, at 16:24, Glyph wrote:
> > On Jul 3, 2020, at 8:01 AM, Maarten ter Huurne  
> > wrote:
> > 
> > Hi all,
> > 
> > I'm happy to announce that a new release of pydoctor, the documentation 
> > extractor used by Twisted, is available on PyPI.
> > 
> > https://pypi.org/project/pydoctor/
> > 
> > Thanks to everyone who contributed with patches and reviews!
> 
> Maarten, we all owe you a significant debt of gratitude for really 
> shepherding PyDoctor into the era of modern Python. Thank you so much for 
> relieving this maintenance burden and making PyDoctor into something I can 
> once again confidently recommend!
> 
> -glyph
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Running tests with monkeytype

[Moshe Zadka]

Hi all,

If you want to get a first rough draft of types for mypy, has anyone tried 
running the tests under monkeytype[1]?

Moshe Z.

[1] https://monkeytype.readthedocs.io/en/stable/___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Moshe Zadka]

May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, 
so maybe we can wait?
I don't have strong feelings, but it is a datapoint.

Moshe Z.

On Wed, May 13, 2020, at 22:13, Glyph wrote:
> On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) 
>  wrote:
> > 
> > On 14/5/20 5:10 am, Craig Rodrigues wrote:
> >> I would like to propose that Twisted drop support for Python 3.5 one year 
> >> from now in 2021.
> > 
> > I was considering proposing dropping it soon (I was thinking when we had 
> > 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
> 
> 
> I'm also inclined to say sooner rather than later; I don't know the numbers 
> but my impression is that 3.x uptake began for real at 3.4 and in earnest at 
> 3.6, so there aren't that many people still on 3.5.
> 
> The lack of variable annotations, in particular, seems like a sticking point 
> with 3.5 as we attempt to adopt mypy.
> 
> -g
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

[Moshe Zadka]

On Mon, Jun 24, 2019, at 17:24, Chengi Liu wrote:
> How do I use more multiple CPU cores? I guess atleast the fact that twisted 
> app wont be blocking is good, but what would be a good way to use multiple 
> cores.

It *really* depends on what is the function that is CPU heavy, and why it is 
CPU heavy.
You have a lot of options here, and they're all subtly different.

But I'll start with noting that this is *no longer* a Twisted question: using 
multiple cores from Python is genuinely a problem for which there is no single 
good solution, just a number of different ones with trade-offs. As I mentioned, 
you can use `dask`. I think that might be the simplest solution for you -- dask 
has a lot of good tutorials, and if your only goal is using multiple CPUs, it 
has an "auto-setup". Another option is to see if you can use a library that 
natively releases the Global Interpreter Lock. If, for example, your function 
is numeric computation heavy, than numpy does release the GIL. You can also use 
ampoule, though the set-up becomes non-trivial -- you will need to either 
change your process initialization or maintain a process pool from the main 
process. Either option can become non-trivial fast. Finally, the option of 
writing the CPU heavy code in Cython, and then releasing the GIL explicitly 
around the area it is not needed, exists.

To sum up:

* Dask
* Something that releases the GIL, like numpy
* Setting up an ampoule process pool
* Move CPU heavy logic to Cython, and release GIL explicitly

As I said, none of these options are really Twisted-related (the only one 
vaguely connected is the ampoule one, but even then, the fact that you'll use 
Twisted to manage the pool is incidental). There are even more exotic options, 
of course, but I think this e-mail might already be confusing enough.

Moshe Z.___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

[Moshe Zadka]

On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote:
>  cpu_res *= yield *defer.*gatherResults*(cpus)

Remember: This will not block the reactor (Good!) but will still limit you to 
one CPU core (in general, some caveats, but true.) If you are CPU bound, this 
is woefully underutilizing modern CPU resources (again, some caveats here).

> if __name__ == '__main__':

In general, the best thing to do is to to use twisted.internet.task.react for 
main. This would also solve the other problem you point out below.

https://twistedmatrix.com/documents/current/api/twisted.internet.task.html#react
 has a good example.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

[Moshe Zadka]

Hi Cheng,

deferToThread *returns* a deferred, and sends the work to a thread pool. The 
other two functions are useful for manipulating deferreds or other promises, 
but will not send work to a separate thread, so they cannot be used instead.

The simple example is pretty simple, it's pretty much

@inlineCallbacks
def some_function():
 d1 = deferToThread(cpu_heavy_square, 5)
 d2 = deferToThread(cpu_heavy_square, 6)
 five_squared, six_squared = yield gatherResults([d1, d2])
 returnValue(five_squared + six_squared)

Note that the tutorial below can be downloaded in notebook form from 
https://github.com/moshez/twisted-tutorial/blob/master/Twisted-Tutorial.ipynb

However, as explained, this will *defer to threads*, but will *not* use 
multiple cores, since Python is single-core. This is not entirely accurate: if 
your heavy functions are using numpy, numpy will release the GIL sometimes. 
Otherwise, you might want to use ampoule, but there are no easy tutorials for 
it that I'm aware of. You might want to consider something like dask + 
deferToThread for using multiple cores. See 
https://github.com/dask/dask-tutorial/blob/master/01_dask.delayed.ipynb for a 
simple example: you just plan your computation, and run the `compute` method in 
a thread. In this case, the only thing the thread is doing is avoiding blocking 
the reactor, while dask internally apportions computations.

(Thank you for the kind of words about the tutorial -- I'm one of the 
presenters.)

Moshe Z.

On Mon, Jun 24, 2019, at 11:34, Chengi Liu wrote:
> Thanks for pointing that out. Is there a dead simply example on how to 
> parallelize a function using ampoule.
> Again, I have a function.. I have a for loop.. the function takes in an 
> argument from loop.. but the for loop is extremely parallelizable.
> 
> Also, I was going thru this awesome twisted tutorial from pycon 2107 
> (https://www.youtube.com/watch?v=ztkBG4qLR_4)
> I found couple of use features.
> (defer.*ensureDeferred*
> and
> defer.*gatherResults*
> )
> Can these two be used instead of deferToThread.
> When are the above defered used and when to use deferToThread.
> 
> Thanks for entertaining my dumb questions.
> 
> On Mon, Jun 24, 2019 at 1:28 AM Gelin Yan  wrote:
>> Hi
>> 
>>  If your purpose is merely not to block the reactor, you can run your code 
>> in a separate thread by using deferToThread. If you want to benefit from 
>> multi cores, you may consider use https://github.com/twisted/ampoule or 
>> other similar tools.
>> 
>> Regards
>> 
>> gelin yan
>> ___
>>  Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Theory and practice of Pull Requests

[Moshe Zadka]

FYI, I opened https://twistedmatrix.com/trac/ticket/9292 which seeks to
realign our theory (what we say in CONTRIBUTING) and what we do in practice
regarding pull requests on GitHub. Please comment on the ticket if you see
any issues!

Moshe Z.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Changing procmon.ProcessMonitor.processes

[Moshe Zadka]

OK I opened a ticket with a plan, after discussing with Mark Williams.

1. Make a custom class that implements the Sequence ABC and pretends to be
sized the old size.
2. Internally all access will be moved to attributes
3. All sequence methods will be marked as Deprecated. We'll kill them in a
year.
4. processes will become _processes, with a Deprecated .processes accessor,
again for backwards compatibility.

I am technically changing the class of a thing from a tuple to a different
class, but I don't think that this can be in general relied on. The API
docs do not mention the attribute, much less promise the concrete class of
its contents.

Objections?

Ticket: https://twistedmatrix.com/trac/ticket/9287

On Wed, Sep 20, 2017 at 1:34 AM ex vito <ex.vitor...@gmail.com> wrote:

> On 2017-09-19, at 15:49, Moshe Zadka <zadka.mo...@gmail.com> wrote:
>
> On Tue, Sep 19, 2017 at 3:51 AM ex vito <ex.vitor...@gmail.com> wrote:
>
>
>
>> Other than that, again, per that section's rules, not being a commiter
>> myself, I'm in no position to approve such a change. I wonder, however, how
>> "urgent" such a final change is to you and why a deprecation cycle does not
>> fit your purpose (even though, admittedly, it may represent more effort).
>>
>
> I think that between the low likelihood that someone went crawling over
> the attributes manually, the RoI of having a deprecation cycle with some
> intermediate solution that later needs to be cleaned up, and the fact that
> this would be a clean break (i.e., "AttributeError") rather than some
> obscure error, I am at least interested in opinions about going the
> exception route.
>
> ProcMon is non-trivial to product ionize, and I'm not aware of anyone even
> using it in production, other than me, let alone crawling over its internal
> state.
>
>
> Agreed. For completeness, the code I work with does not make use of
> ProcMon and I don't recall having ever used it.
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How to correctly run Sqlite with Twisted?

[Moshe Zadka]

The best answer is probably https://github.com/alex/alchimia

On Tue, Sep 19, 2017 at 1:59 PM Goffi  wrote:

> Hello,
>
> I'm using Sqlite3 module through Twisted's enterpirse.adbapi, I create the
> ConnectionPool instance like this:
>
>self.dbpool = ConnectionPool("sqlite3", db_filename,
> check_same_thread=False)
>
> You can see the code at
> https://repos.goffi.org/sat/file/tip/src/memory/sqlite.py
>
> Sometime, the writing is failing with following exception:
>
>   Failure instance: Traceback: :
> database is
> locked
>
> So I wonder if the database is correctly used, did anybody experienced
> something similar with Twisted and Sqlite ?
>
> Should I just augment timeout as advised at https://stackoverflow.com/a/
> 8618328? Looks like more a workaround than a clean solution.
>
> Python 2 documentation doesn't talk about check_same_thread argument, but
> Python 3 at
> https://docs.python.org/3.5/library/sqlite3.html#sqlite3.connect
> says that writing operation should be serialized by the user (I thought it
> was
> the default as said in https://sqlite.org/threadsafe.html), how should I
> achieve that?
>
> Also PRAGMA are not working (specially "PRAGMA foreign_keys = ON"), I guess
> because of multi-threading, what is the good way to activate foreign_keys
> for
> all following request?
>
> Thanks in advance
>
> Goffi
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Changing procmon.ProcessMonitor.processes

[Moshe Zadka]

On Tue, Sep 19, 2017 at 3:51 AM ex vito  wrote:


> Other than that, again, per that section's rules, not being a commiter
> myself, I'm in no position to approve such a change. I wonder, however, how
> "urgent" such a final change is to you and why a deprecation cycle does not
> fit your purpose (even though, admittedly, it may represent more effort).
>

I think that between the low likelihood that someone went crawling over the
attributes manually, the RoI of having a deprecation cycle with some
intermediate solution that later needs to be cleaned up, and the fact that
this would be a clean break (i.e., "AttributeError") rather than some
obscure error, I am at least interested in opinions about going the
exception route.

ProcMon is non-trivial to productionize, and I'm not aware of anyone even
using it in production, other than me, let alone crawling over its internal
state.

Moshe Z.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Changing procmon.ProcessMonitor.processes

[Moshe Zadka]

On Mon, Sep 11, 2017 at 2:44 AM ex vito  wrote:

> Sidenote: A quick, mostly backwards compatible, change could probably add
> cwd to the existing per-process tuple in self.processes as an extra item,
> couldn't it?


No. Most usage of this tuple is unpacking, which makes it really really not
backwards compatible.


> > 4. Decide we're going with "incompatible change", and move processes to
> a private attribute.
>
> I see no reason for accessing those attributes from "outside of
> ProcessMonitor" but if Twisted claims not to break existing code without
> previous deprecation warning, such an approach would break that promise
> (full policy details here
> http://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html
> ).
>

Specifically, I was asking about
http://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html#procedure-for-exceptions-to-this-policy
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Changing procmon.ProcessMonitor.processes

[Moshe Zadka]

Hello all,

For context, please read bullet one of Glyph's comment on
https://twistedmatrix.com/trac/ticket/3691#comment:9

I am in a similar situation to JP 8 years ago -- I want to add a working
directory argument to addProcess, so that processes that care (usually for
silly reasons) about their cwd can be monitored.

The options I see:

1. Move processes to a private attribute, deprecate processors, then do it.
2. Have a parallel dictionary to processes
3. Move, as Glyph suggested, to object with attributes. For backwards
compatibility, I can support __getitem__ for the historic unpackers.
4. Decide we're going with "inccompatible change", and move processes to a
private attribute.

Anyone wants to weigh in?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Proposal -- Code of Conduct

[Moshe Zadka]

Hi everyone,

I'm sorry I haven't responded to the e-mails -- I have had a busy weekend,
and my stack was a bit deep for personal projects. I will try to get caught
up and see what the right next step is on Monday.

Thanks for your understanding,
Moshe Z.

On Wed, Jun 24, 2015 at 1:27 PM Kevin Turner ket...@keturn.net wrote:

  I'm generally +1 on the idea of having a CoC.

 I'm content to delegate the details to subcommittee, so maybe I should
 leave it at that, but I find there's one point I want to reinforce here:


 On Sun, Jun 21, 2015, at 08:46 AM, Moshe Zadka wrote:

 4. A CoC has several purposes, and banning people is actually pretty far
 down the list. […]


 Agreed. Clayton said They rarely get read until there's a problem, but I
 think prevention and expectation-setting are things to aim for here. If
 people only find out about what's acceptable conduct *after* there's a
 problem, that's too late.



 and as an aside,


 I think it's a problem that we don't have the names of the people on the
 PLC on our wiki, but this is a separate issue. [It should be rectified, as
 well as clarifying the rules for how the PLC acquires/retires members.


 Agreed. (I am one of the current PLC members.)
  ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Proposal -- Code of Conduct

[Moshe Zadka]

Some repilies to all the issues mentioned here:

1. I think we should definitely maintain our own copy. Sorry for it not
being a fork, the only non-formatting changes I made are things that
referrred to the Django-run events (since we don't run events -- I did
replace some of this language with mention of the meetup group). As for
CC-BY: I did refer to the Django CoC, I thought CC-BY only refers to the
last point of copying. If it makes sense to add Speak Up! to the mention
for license compliance, I'm happy to do that.

2. The Django CoC specifically has a change procedure. I did not copy that
part over, since our organization is a lot less formal -- I'm pretty sure
it's if the PLC agrees. I think it's a problem that we don't have the
names of the people on the PLC on our wiki, but this is a separate issue.
[It should be rectified, as well as clarifying the rules for how the PLC
acquires/retires members.

3. I considered briefly the PSF one. I think it is *way* under-specified. I
want to have something that's clear. I think Django, specifically, has a
pretty good track record of promoting diversity and the PSF has a...worse
one.

4. A CoC has several purposes, and banning people is actually pretty far
down the list. The first reason is that we assume most people are good
people, or at least wish to follow the rules, and having clear rules about
what's not acceptable will hopefully lead to people voluntarily complying.
The second reason is to send out a message to underrepresented groups: we
value you, and will protect your right to be treated with dignity in our
community. While this does not solve diversity problems by itself, it is
an important step. A distant third, in my opinion, is the banning of people
-- not least because this issue has not really come up [some people got
banned from IRC, I believe, but usually not for the kind of reasons
described in the CoC, and I don't remember people getting banned from Trac
or the ML]. I really did mean what I said in my first e-mail: we do try to
be an open and inclusive place. We are *already* complying with the CoC.
[If you subscribe to a Hyackean view of law and legislation, this is law,
not legislation.]

5. Not a reply to an e-mail, but I like to publicly call out Hyneck for
actually reading through the CoC, noticing a couple of mistakes and sending
me a pull-request. It is much appreciated!

Thanks everyone who took the time to read and think about this important
issue,
Moshe Z.

On Sun, Jun 21, 2015 at 4:38 AM anatoly techtonik techto...@gmail.com
wrote:

 On Sun, Jun 21, 2015 at 2:30 PM, Amber Hawkie Brown
 hawk...@atleastfornow.net wrote:
 
  On 21 Jun 2015, at 19:00, Hynek Schlawack h...@ox.cx wrote:
 
  I am sure everyone understands that the Twisted community would love
 more diversity. While it is hard to achieve, it should be easy to remove
 one of the obvious blockers -- making underrepresented groups feel more
 welcome.
  Thanks for taking this on, Moshe.
 
  +1
 
  My current draft, including instructions on how to build it, is in
 https://github.com/moshez/twisted-coc . I have intentionally not made the
 built documents available, in an attempt to avoid someone picking them up
 before they're approved by us.
 
  Why isn't this repository either (A) just a simple text file saying
 we have adopted the Django CoC or (B) a very small fork of something
 else?  One of the concerns is licensing; if the text comes via Django,
 Django credits the Speak Up! project, which is CC-BY, apparently from
 this repository: https://github.com/jnoller/talk-mentorship.  Another
 is... is Twisted really distinct enough to need its own CoC?  Just
 s/Django/Twisted might be good enough?  (Since this is not a fork, figuring
 out if anything else has changed is rather tedious, even after having read
 both ;)).
 
  I wonder whether it might make sense to just say we adopt
 https://www.python.org/psf/codeofconduct/ ?
 
  What I would really love is if we could have our own diversity
 statement like Django has: https://www.djangoproject.com/diversity/
 
  The Django one is more explicit -- it's rather sad that it needs to be,
 but it does lay out more directly the unacceptable behaviours.
 
  That being said, it's not an unchangeable document -- if it doesn't suit
 the community's needs, we can modify it to suit.

 So that CoCs are just a set of rules to ban users expressed in a vague
 legal form, so that people can not complain, because they don't
 understand. Is that right? =)

 --
 anatoly t.

 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Proposal -- Code of Conduct

[Moshe Zadka]

Hi everyone,

I am sure everyone understands that the Twisted community would love more
diversity. While it is hard to achieve, it should be easy to remove one of
the obvious blockers -- making underrepresented groups feel more welcome.

I think, and hope, that our IRC channel, our issue system and mailing list
have been a friendly, pleasant place. This is an attempt to clarify what we
mean by a friendly, pleasant place.

After some discussion on IRC, I volunteered to write up a Code of Conduct
for Twisted. It is mostly an adaptation of Django's CoC -- I think Django
has a nice track record of commitment to diversity, and, of course, we
expect our communities to overlap.

My current draft, including instructions on how to build it, is in
https://github.com/moshez/twisted-coc . I have intentionally not made the
built documents available, in an attempt to avoid someone picking them up
before they're approved by us.

Please respond if there are any concerns about the wording, anything that I
missed and anything you think does not belong there. I hope we can achieve
consensus, and have the Project Leadership Committee approve this
(including approving the current committee -- I've volunteered to chair it,
and Glyph and Amber (HawkOwl) have graciously agreed to be on it.)

Thanks,
Moshe Z.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] INCOMPATIBLE CHANGE: Adding parent as a required attribute to IService

[Moshe Zadka]

On Thu, Jun 4, 2015 at 11:50 AM Glyph gl...@twistedmatrix.com wrote:

 I'm in favor of this change

...

  Other non-inheriting IService implementations have it too, at least from
 what I have heard on IRC so far.


I haven't heard anything on IRC, but maybe you're thinking of Tristan's
e-mail here:
https://twistedmatrix.com/pipermail/twisted-python/2015-May/029521.html
where he said they did, indeed, have a parent.

Thanks,
Moshe Z.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] INCOMPATIBLE CHANGE: Adding parent as a required attribute to IService

[Moshe Zadka]

IService's docstrings obliquely referred to a parent attribute, but have
not actually specified its type or existence. This change makes this
attribute mandatory. It also makes the existence of the other IService
attributes, running and name, enforced by verifyObject.

The ticket is available here: https://twistedmatrix.com/trac/ticket/7922
and the buildbots are green on the branch (the intermittent thread failure
failed intermittently).

Especially if you defined your own IService implementations that do not
inherit from Service, please run your unittests with this branch.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[Moshe Zadka]

Hi glyph and exarkun,

[1] The compose syntax is the least interesting part for me here, and seems
to cause the most bike-shedding, so I think I will punt on that. Instead, I
will write a function called compose() which can be used from .tac files
or possibly other plug-ins, leaving the issue of the syntax for a different
day.
[2] It seems like the biggest inconsistency in the docstrings is that they
mention the parent without having an attribute for it. (E.g., the docstring
for setServiceParent *says* parent attribute is set, but there's no
parent attribute in the interface. Glyph, if you want to open a ticket
with the issues you found, please assign it to me -- otherwise, more
specificity is useful!
[2a] Since the docstrings do refer to the parent attribute, and since I
would be surprised if services just implemented the interface without
inheriting from Service, I wonder if people object *too* much if I fix
it by documenting the parent attribute as part of the interface?
[3] I think that with the compose() function available in .tac files, maybe
needing .tac *with* subcommands is not quite as important? Maybe I'll leave
that off of the initial sprint.

So bottom line, current plans:
[4] Implement compose() as a function that takes an iterable-of-iterables,
treats them as command-line arguments for services, and builds a
MultiService() with them all.
[5] Document the parent attribute in IService (and possibly other issues,
depending on Glyph opening a bug for [2]
[6] Implement provides() the way I wanted to (now that parent is documented)

Since it seems like compose-as-a-function is the least controversial bit
here, I'll start with that while the other issues settle down.

Moshe Z.


On Wed, May 20, 2015 at 5:18 PM exar...@twistedmatrix.com wrote:

 On 07:29 pm, gl...@twistedmatrix.com wrote:
 
 On May 20, 2015, at 04:13, exar...@twistedmatrix.com wrote:
 
 On 19 May, 09:01 pm, tom.pri...@ualberta.net wrote:
 Glyph gl...@twistedmatrix.com writes:
 You could also find some other way to split the argument list but
 + doesn't seem particularly obscure in this context to me.  (If
 there's really a need to pass a literal + to a plugin we could add
 an escaping syntax as well.)
 
 I think if we are adding syntax, then we should also add escaping at
 the
 same time.
 
 On a related note, when designing this kind of syntax, I think it is
 often valuable to explictly leave some of the space as an explict
 error,
 to leave freedom to extend the syntax in the future.
 
 I think this is 100% correct.  This is part of why I don't want the
 syntax added to `Options`.  If it's a feature of, say, a compose
 twistd plugin then you can always throw the whole compose twistd
 plugin in the trash and start again.  That gives you quite a lot of
 space for syntax changes. :)
 
 (And of course, not introducing a syntax at all leaves you even more
 room... but talking this crowd out of inventing weird syntaxes is
 probably an exercise in futility.)
 
 Much as I love weird syntaxes, using + as a separator for a 'compose'
 plugin seems about as straightforward as I could imagine such a thing
 working.  Do you have an alternate proposal that is less 'weird'?  I
 wouldn't promise to accept it but I wouldn't want to go with something
 unnecessarily weird just due to a failure of my imagination.

 It's possible this is a case of necessary weirdness - at least, if
 you're dead-set on some kind of option-based CLI interface to this
 functionality.

 Think about other CLI tools that try to offer this kind of composition.
 Only two really come to mind for me and one of those is not exactly well
 regarded for its novel syntax (hint - it rhymes with schmestreamer).

 This is not to say that this kind of composition is a bad thing - or
 even that no other widely-used software supports the kind of invoke-
 several-things-of-stuff behavior that we're talking about providing
 access to here.

 However, I do think it is the case that most of the world tries to solve
 this problem without trying to force everything into a world-view framed
 by GNU getopt().

 With that in mind, here are some alternatives:

   (0) The null proposal

 $ twistd compose \
 web --port 80 +
 manhole --auth foo + \
 logging --path /var/run

   (1) Borrow `--` which already means stop parsing options here.  It
 makes sense if you think about sub-commands as existing on a stack and
 `--` meaning pop the stack.  Possibly not actually feasible since I
 think `Options` already knows what `--` means (by virtue of actually
 *using* getopt()) and uses it to separate options from positional
 arguments (and therefore it's not an unambiguous separator - but I
 haven't double-checked this).

 $ twistd compose \
 web --port 80 -- \
 manhole --auth foo -- \
 logging --path /var/run

   (2) Put things into strings which already have well-defined escape
 syntax:

 $ twistd compose \
 'web --port 80' \
 

Re: [Twisted-Python] Multiple plugins in twistd

[Moshe Zadka]

FYIFYI --  Patch attached. Docs are missing, but I'm sure JP/Glyph will be
happy to tell me what I've done wrong otherwise :-P

On Wed, May 20, 2015 at 7:40 PM Moshe Zadka zadka.mo...@gmail.com wrote:

 FYI, https://twistedmatrix.com/trac/ticket/7907#ticket created.

 On Wed, May 20, 2015 at 7:31 PM Moshe Zadka zadka.mo...@gmail.com wrote:

 Hi glyph and exarkun,

 [1] The compose syntax is the least interesting part for me here, and
 seems to cause the most bike-shedding, so I think I will punt on that.
 Instead, I will write a function called compose() which can be used from
 .tac files or possibly other plug-ins, leaving the issue of the syntax for
 a different day.
 [2] It seems like the biggest inconsistency in the docstrings is that
 they mention the parent without having an attribute for it. (E.g., the
 docstring for setServiceParent *says* parent attribute is set, but
 there's no parent attribute in the interface. Glyph, if you want to open
 a ticket with the issues you found, please assign it to me -- otherwise,
 more specificity is useful!
 [2a] Since the docstrings do refer to the parent attribute, and since I
 would be surprised if services just implemented the interface without
 inheriting from Service, I wonder if people object *too* much if I fix
 it by documenting the parent attribute as part of the interface?
 [3] I think that with the compose() function available in .tac files,
 maybe needing .tac *with* subcommands is not quite as important? Maybe I'll
 leave that off of the initial sprint.

 So bottom line, current plans:
 [4] Implement compose() as a function that takes an
 iterable-of-iterables, treats them as command-line arguments for services,
 and builds a MultiService() with them all.
 [5] Document the parent attribute in IService (and possibly other issues,
 depending on Glyph opening a bug for [2]
 [6] Implement provides() the way I wanted to (now that parent is
 documented)

 Since it seems like compose-as-a-function is the least controversial bit
 here, I'll start with that while the other issues settle down.

 Moshe Z.


 On Wed, May 20, 2015 at 5:18 PM exar...@twistedmatrix.com wrote:

 On 07:29 pm, gl...@twistedmatrix.com wrote:
 
 On May 20, 2015, at 04:13, exar...@twistedmatrix.com wrote:
 
 On 19 May, 09:01 pm, tom.pri...@ualberta.net wrote:
 Glyph gl...@twistedmatrix.com writes:
 You could also find some other way to split the argument list but
 + doesn't seem particularly obscure in this context to me.  (If
 there's really a need to pass a literal + to a plugin we could add
 an escaping syntax as well.)
 
 I think if we are adding syntax, then we should also add escaping at
 the
 same time.
 
 On a related note, when designing this kind of syntax, I think it is
 often valuable to explictly leave some of the space as an explict
 error,
 to leave freedom to extend the syntax in the future.
 
 I think this is 100% correct.  This is part of why I don't want the
 syntax added to `Options`.  If it's a feature of, say, a compose
 twistd plugin then you can always throw the whole compose twistd
 plugin in the trash and start again.  That gives you quite a lot of
 space for syntax changes. :)
 
 (And of course, not introducing a syntax at all leaves you even more
 room... but talking this crowd out of inventing weird syntaxes is
 probably an exercise in futility.)
 
 Much as I love weird syntaxes, using + as a separator for a 'compose'
 plugin seems about as straightforward as I could imagine such a thing
 working.  Do you have an alternate proposal that is less 'weird'?  I
 wouldn't promise to accept it but I wouldn't want to go with something
 unnecessarily weird just due to a failure of my imagination.

 It's possible this is a case of necessary weirdness - at least, if
 you're dead-set on some kind of option-based CLI interface to this
 functionality.

 Think about other CLI tools that try to offer this kind of composition.
 Only two really come to mind for me and one of those is not exactly well
 regarded for its novel syntax (hint - it rhymes with schmestreamer).

 This is not to say that this kind of composition is a bad thing - or
 even that no other widely-used software supports the kind of invoke-
 several-things-of-stuff behavior that we're talking about providing
 access to here.

 However, I do think it is the case that most of the world tries to solve
 this problem without trying to force everything into a world-view framed
 by GNU getopt().

 With that in mind, here are some alternatives:

   (0) The null proposal

 $ twistd compose \
 web --port 80 +
 manhole --auth foo + \
 logging --path /var/run

   (1) Borrow `--` which already means stop parsing options here.  It
 makes sense if you think about sub-commands as existing on a stack and
 `--` meaning pop the stack.  Possibly not actually feasible since I
 think `Options` already knows what `--` means (by virtue of actually
 *using* getopt()) and uses it to separate options from

Re: [Twisted-Python] Multiple plugins in twistd

[Moshe Zadka]

FYI, https://twistedmatrix.com/trac/ticket/7907#ticket created.

On Wed, May 20, 2015 at 7:31 PM Moshe Zadka zadka.mo...@gmail.com wrote:

 Hi glyph and exarkun,

 [1] The compose syntax is the least interesting part for me here, and
 seems to cause the most bike-shedding, so I think I will punt on that.
 Instead, I will write a function called compose() which can be used from
 .tac files or possibly other plug-ins, leaving the issue of the syntax for
 a different day.
 [2] It seems like the biggest inconsistency in the docstrings is that they
 mention the parent without having an attribute for it. (E.g., the docstring
 for setServiceParent *says* parent attribute is set, but there's no
 parent attribute in the interface. Glyph, if you want to open a ticket
 with the issues you found, please assign it to me -- otherwise, more
 specificity is useful!
 [2a] Since the docstrings do refer to the parent attribute, and since I
 would be surprised if services just implemented the interface without
 inheriting from Service, I wonder if people object *too* much if I fix
 it by documenting the parent attribute as part of the interface?
 [3] I think that with the compose() function available in .tac files,
 maybe needing .tac *with* subcommands is not quite as important? Maybe I'll
 leave that off of the initial sprint.

 So bottom line, current plans:
 [4] Implement compose() as a function that takes an iterable-of-iterables,
 treats them as command-line arguments for services, and builds a
 MultiService() with them all.
 [5] Document the parent attribute in IService (and possibly other issues,
 depending on Glyph opening a bug for [2]
 [6] Implement provides() the way I wanted to (now that parent is
 documented)

 Since it seems like compose-as-a-function is the least controversial bit
 here, I'll start with that while the other issues settle down.

 Moshe Z.


 On Wed, May 20, 2015 at 5:18 PM exar...@twistedmatrix.com wrote:

 On 07:29 pm, gl...@twistedmatrix.com wrote:
 
 On May 20, 2015, at 04:13, exar...@twistedmatrix.com wrote:
 
 On 19 May, 09:01 pm, tom.pri...@ualberta.net wrote:
 Glyph gl...@twistedmatrix.com writes:
 You could also find some other way to split the argument list but
 + doesn't seem particularly obscure in this context to me.  (If
 there's really a need to pass a literal + to a plugin we could add
 an escaping syntax as well.)
 
 I think if we are adding syntax, then we should also add escaping at
 the
 same time.
 
 On a related note, when designing this kind of syntax, I think it is
 often valuable to explictly leave some of the space as an explict
 error,
 to leave freedom to extend the syntax in the future.
 
 I think this is 100% correct.  This is part of why I don't want the
 syntax added to `Options`.  If it's a feature of, say, a compose
 twistd plugin then you can always throw the whole compose twistd
 plugin in the trash and start again.  That gives you quite a lot of
 space for syntax changes. :)
 
 (And of course, not introducing a syntax at all leaves you even more
 room... but talking this crowd out of inventing weird syntaxes is
 probably an exercise in futility.)
 
 Much as I love weird syntaxes, using + as a separator for a 'compose'
 plugin seems about as straightforward as I could imagine such a thing
 working.  Do you have an alternate proposal that is less 'weird'?  I
 wouldn't promise to accept it but I wouldn't want to go with something
 unnecessarily weird just due to a failure of my imagination.

 It's possible this is a case of necessary weirdness - at least, if
 you're dead-set on some kind of option-based CLI interface to this
 functionality.

 Think about other CLI tools that try to offer this kind of composition.
 Only two really come to mind for me and one of those is not exactly well
 regarded for its novel syntax (hint - it rhymes with schmestreamer).

 This is not to say that this kind of composition is a bad thing - or
 even that no other widely-used software supports the kind of invoke-
 several-things-of-stuff behavior that we're talking about providing
 access to here.

 However, I do think it is the case that most of the world tries to solve
 this problem without trying to force everything into a world-view framed
 by GNU getopt().

 With that in mind, here are some alternatives:

   (0) The null proposal

 $ twistd compose \
 web --port 80 +
 manhole --auth foo + \
 logging --path /var/run

   (1) Borrow `--` which already means stop parsing options here.  It
 makes sense if you think about sub-commands as existing on a stack and
 `--` meaning pop the stack.  Possibly not actually feasible since I
 think `Options` already knows what `--` means (by virtue of actually
 *using* getopt()) and uses it to separate options from positional
 arguments (and therefore it's not an unambiguous separator - but I
 haven't double-checked this).

 $ twistd compose \
 web --port 80 -- \
 manhole --auth foo

[Twisted-Python] Multiple plugins in twistd

[Moshe Zadka]

Background

Currently, twistd assumes one-run-one-plugin. It would be nice to load up
multiple plug-ins in Twisted, for many reasons. These include: serving the
same in-memory content via separate protocols, adding manhole to other
plug-ins (so the end-deployer can add it to other things, as opposed to the
original implementor, and a catch-all category of auxiliary services.

Auxiliary services are those which are not useful in and of themselves, but
add value to a service which does something else of use.

Examples of auxiliary services -- a logging service (that connects to some
logging protocol on start-up), a metrics service (that sends statistics to
a collector like statsd or riemann) or an error-sending service (to
something like Sentry).

Proposal

tl;dr: four new tickets (codenamed SUBCOMMANDS, SERVICES, MANHOLE and
PROVIDERS) and one old ticket (3538)

SUBCOMMANDS: Add '+' as a special character in t.p.usage.Options. This
behavior will be off by default, and controlled by an attribute on the
Options instance allowMultipleSubcommands.

The attribute will only be checked when the first sub-command starts, to
allow setting it based on global flags. When the option is on, '-+' will be
passed as '+' to the Options instance, to allow sending plain '+' to
sub-commands.

Glyph thinks there's a ticket for it. I couldn't find it in search for
tickets in 'core' whose description mentionds 'command'. Unless anyone can
find it, I'll open a new ticket.

SERVICES (depends on SUBCOMMANDS): In twistd, set the flag aMS if
'--allow-multiple-services' is given. Add to the application all services.

3538 (depends on SERVICES): If '--allow-multiple-services' is given, and
'--python .tac file' is given, process subcommands as usual.

PROVIDERS: Add a function,
   providersInHierarchy(IService, IInterface) - List[IInterface]
that returns all services in the hierarchy which provide the interface.
This ticket does not depend on any other tickets.

MANHOLE (depends on PROVIDERS, SERVICES): Add a built-in twistd plugin
named manhole. The plugin will expose manhole as PB/telnet with a
namespace that includes
  {'services': providersInHierarchy(manholeService, IService)}
This ticket technically could only depend on PROVIDERS, but to be useful,
it also depends on SERVICES
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Multiple plugins in twistd

[Moshe Zadka]

OK, so let me once again enumerate the tickets I'm going to open, because
it seems like we're reaching consensus:

[1] PROVIDES (pretty much as written: exarkun -- the idea is to walk up the
hierarchy and then back down, so I did mean IService -- go up until you
find a root, then go down and get all descendents)
[2] 3538 -- will do it with a flag to turn on .tac + plugin
[2.1] I'm assuming the deprecation dance is flag to turn on - flag to
turn on + warn if .tac+plugin and no flag - make flag no-op - warn on
flag - remove flag. If someone doesn't want this dance, please let me
know preferred alternatives.
[3] MAKE -- API for createServiceFromNameAndOptions (basically,
compose-as-a-library)
[4] COMPOSE (implements twistd_compose plugin)

I think there's at least rough consensus that these four pieces are useful,
even if there's still some disagreement on the details, so maybe the next
step is to open the three to-be-opened tickets, and then to discuss the
details on the tickets. I'm going to do the ticket opening tomorrow, and
maybe even work on them during the SF Python Meetup (anyone in SF -- you
should [a] go and [b] say hi also [c] optionally, help me with that).

If anyone has serious objections to this plan, let me know!

Thanks,
Moshe Z.

On Tue, May 19, 2015 at 2:35 PM Glyph gl...@twistedmatrix.com wrote:


  On May 19, 2015, at 14:01, Tom Prince tom.pri...@ualberta.net wrote:
 
  Glyph gl...@twistedmatrix.com writes:
 
  You could also find some other way to split the argument list but +
 doesn't seem particularly obscure in this context to me.  (If there's
 really a need to pass a literal + to a plugin we could add an escaping
 syntax as well.)
 
  I think if we are adding syntax, then we should also add escaping at the
  same time.
 
  On a related note, when designing this kind of syntax, I think it is
  often valuable to explictly leave some of the space as an explict error,
  to leave freedom to extend the syntax in the future.
 
  I wonder if there is any value in having a syntax that is nestable. I
  don't see any specific use case, but I can imagine wanting ot pass
  options to `compose` that are scoped to an individual plugin that it is
  loading. And then, maybe you want to nest things so that those options
  apply to a subset of plugins, which might naturally be implemented as
  compos of a compose.
 
  The suggestions in the last paragraph are perhaps somewhat contrived,
  and certainly not something that should be *implemnted* in a first (or
  even second pass). But considering the possiblity, and then picking a
  syntax that might allow those kinds of extension (and then explictly
  making the extension syntax an error) gives us the ability to add those
  features without breaking backwards compatibility.

 We can un-escape '\+' as a token just fine, and if we do that, all of the
 weird use-cases you just described are possible.  I can't think of any
 options I'd want to pass to 'compose' myself, but it would be easy enough
 to add some flags to its parser.

 -g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python