Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

2020-05-22 Thread Glyph 


> On May 19, 2020, at 11:14 AM, Tom Most  wrote:
> 
> On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
>> On Tue, May 19, 2020 at 8:33 AM Barry Scott > > wrote:
>> Why not waitpid for all children (pid=0 or pid=-1) in a single call? 
>> 
>> Because then you might reap a child process someone launched with a 
>> different API - eg the Python subprocess module.
> 
> Exactly. I am currently using this as a workaround, but it's not a general 
> solution. You need to be careful because subprocess use hides in many places. 
> For example, beware uuid.uuid1().

So, pidfd's cool, we should totally use it.  Also we should use posix_spawn and 
maybe some other stuff too.  But I wonder if there's any heuristic we could use 
to speed up our current strategy, like ordering the to-reap list by putting 
things with no open FDs at the front of it?  And optimistically assuming that 
once we've found something to reap, maybe we can stop?  And maybe it should run 
in a cooperator, rather than just blocking the reactor indefinitely?  Honestly 
it had not occurred to me that people were managing 20k+ python interpreters at 
a time with spawnProcess (although, yikes, amazing, you've gotta talk more 
about this application, and what kind of hardware you're on!)

Anyway there's lot of exciting room for performance improvements here, it would 
be nice to have some benchmarks for this kind of high-load process-monitor 
application.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

2020-05-22 Thread Glyph 


> On May 19, 2020, at 5:41 AM, Jean-Paul Calderone  
> wrote:
> 
> On Tue, May 19, 2020 at 8:33 AM Barry Scott  > wrote:
> On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
> 
> > 
> > If looking improve process management in Twisted, I think that the
> > lowest-hanging fruit is to watch children with pidfd, rather than reaping
> > all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> > running process --- starts to consume substantial CPU. IIRC it pegged a
> > core at 20,000 processes or so.
> 
> Why not waitpid for all children (pid=0 or pid=-1) in a single call?
> 
> Barry
> 
> Because then you might reap a child process someone launched with a different 
> API - eg the Python subprocess module.

In case it is not clear from context - we used to do exactly this.  It broke a 
lot of stuff.  The current approach is slow, but robust.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

2020-05-22 Thread Glyph 

> On May 19, 2020, at 1:52 AM, Richard van der Hoff  wrote:
> 
> On 16/05/2020 06:56, Glyph wrote:
>> 
>> 
>>> On May 15, 2020, at 8:40 PM, Craig Rodrigues >> > wrote:
>>> 
>>> Maybe it would be OK to do one more release of Twisted and announce that as 
>>> the last release supporting Python 3.5, before
>>> dropping support?  
>> 
>> Yeah; whenever we drop a Python version we should always support at least 
>> one more release, so that people have some notice before they lose access to 
>> the next set of security updates.
>> 
>> Any 3.5 users on this list who would want to postpone it longer than this?
>> 
> Sadly we have an important customer whose servers run debian oldstable, which 
> means we need to stay compatible with 3.5 until we can persuade them to 
> upgrade, and it's taken a couple of years to get them off python 2.7...
> 
> I'm not sure that should necessarily affect your plans, but I doubt we're 
> alone in this situation.
> 

I guess one thing I'm curious about is why your application would need to be 
installed along with the system Python on those OS versions?  It seems like a 
packaging strategy that ignored the fossilized versions that Debian packages 
with the system and just built its own Python would be more reliable and allow 
for upgrading at least most Python dependencies well beyond what the system 
would allow by policy.  Or, for that matter, why not just run in a Docker 
container?

Matrix is a pretty big user, and so in some sense I care about this specific 
case, but I also find the general question interesting, because I have 
difficulty reasoning about how long to support older versions of things in the 
modern application packaging environment where containers, virtualenvs, and 
associated tooling make it possible to effectively ignore the base environment. 
When & why do you have to pay attention to it?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

2020-05-22 Thread Sereysethy TOUCH 
Hi,

After I tested it on Linux, everything works fine. So I think the problem
is related to MacOS restriction. I already reported the problem to the
maintainers, because the library is supposed to work on both Linux and
MacOS. But this new thread is killing the twisted main reactor.

By the way I have another question regarding ssh client. I developed an ssh
client, after authentication is made, I dont want to open a channel
directly in serviceStarted in connection, I want to keep the connection so
that I can use to open channel for doing exec later, my question is where
should I keep that connection? Currently in my transport I add a defer in
which it will be later called to create another object that I control, the
defer is called back in serviceStarted, which I pass the connection object.
I am not sure what is a correct way to do it. For now I feel like it is a
cycling reference: transport -> connection -> transport -> my object to
control connection.

Thanks,
TS

On Fri, May 22, 2020 at 6:16 PM Barry Scott 
wrote:

> On Thursday, 21 May 2020 21:45:44 BST Sereysethy TOUCH wrote:
> > I am on MacOS, it is a development phase, but deployment will be on
> Linux.
> >
> > It is hard to run dtrace/dtruss on MacOS due to the "system integrity
> > protection" on MacOS. I got some output but very limited information.
> >
> > So if there is a workaround you might suggest?
>
> Why bother debugging it on macOS it never has to work there?
>
> If you are targetting linux then develop and test in linux I'd recommend.
> I use VMware fusion on the mac and run VMs for the target
> environements I need. Works very well.
>
> Barry
>
> >
> > TS
> >
> > On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
> >
> > exar...@twistedmatrix.com> wrote:
> > > On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
> > >
> > > touch.sereyse...@gmail.com> wrote:
> > >> Hi Jean-Paul,
> > >>
> > >> I found the error after working on a short, self contained, correcte
> > >> example. It is not about twisted and normal thread in general, but
> > >> something else.
> > >>
> > >> After more tests, I come to realise that whenever I instantiate this
> > >> class PolicyClient from this library
> > >>
> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py
> > >> in my protocol or just anywhere, twisted app either freezes or
> > >> terminates, and this *only* happens when the app runs as a daemon,
> > >> created using a service (.tac file).
> > >>
> > >> I hope you can take a look at the class PolicyClient, why it causes
> > >> problem to reactor main loop. Please point me to where the idea might
> > >> come
> > >> from.
> > >>
> > >> I can give you the example but not sure if it is enough, as the client
> > >> needs to connect to a server.
> > >
> > > It looks like PolicyClient does a blocking HTTP request in __init__.
> > > Since PolicyClient is instantiated in the reactor thread, this will
> block
> > > the reactor.  Perhaps this call is hanging for some reason when the
> > > process
> > > has daemonized?  This would be unusual but not unheard of.  For
> example,
> > > there are macOS environments where a process does not have access to
> the
> > > network if it is not associated with a windowing session.
> > >
> > > What platform do you observe the problem on, and what more can you
> learn
> > > about what the process does on its way to hanging (eg can you run
> strace
> > > on
> > > it)?
> > >
> > > Jean-Paul
> > >
> > >> Thank you,
> > >> TS
> > >>
> > >>
> > >> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
> > >>
> > >> exar...@twistedmatrix.com> wrote:
> > >>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
> > >>>
> > >>> touch.sereyse...@gmail.com> wrote:
> >  Hello,
> > 
> >  I am developing a twisted app, and it runs as a service using
> twistd -y
> >  to start the app.
> >  I am having a problem of running a thread. I know it is not
> recommended
> >  to use thread, but the library that I use, the object created is
> >  running in
> >  a thread.
> > 
> >  Here is the problem:
> > 
> >  1) if I start reactor by running reactor.run() directly, thread is
> >  running fine
> >  2) if I run it as a service using twisted, the thread is not
> running,
> >  it runs but it seems to be blocked, because I tried to write
> something
> >  to
> >  file using time.sleep(), but file is empty.
> > 
> >  Is there something that I miss? How can I debug this?
> > >>>
> > >>> A good next step would be to create at Short, Self Contained, Correct
> > >>> (Compilable), Example  and share it.
> > >>>
> > >>> Jean-Paul
> > >>>
> >  Thank you,
> > 
> >  TS
> >  ___
> >  Twisted-Python mailing list
> >  Twisted-Python@twistedmatrix.com
> >  https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > >>>
> > >>>

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

2020-05-22 Thread Barry Scott 
On Thursday, 21 May 2020 21:45:44 BST Sereysethy TOUCH wrote:
> I am on MacOS, it is a development phase, but deployment will be on Linux.
> 
> It is hard to run dtrace/dtruss on MacOS due to the "system integrity
> protection" on MacOS. I got some output but very limited information.
> 
> So if there is a workaround you might suggest?

Why bother debugging it on macOS it never has to work there?

If you are targetting linux then develop and test in linux I'd recommend.
I use VMware fusion on the mac and run VMs for the target
environements I need. Works very well.

Barry

> 
> TS
> 
> On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
> 
> exar...@twistedmatrix.com> wrote:
> > On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
> > 
> > touch.sereyse...@gmail.com> wrote:
> >> Hi Jean-Paul,
> >> 
> >> I found the error after working on a short, self contained, correcte
> >> example. It is not about twisted and normal thread in general, but
> >> something else.
> >> 
> >> After more tests, I come to realise that whenever I instantiate this
> >> class PolicyClient from this library
> >> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py
> >> in my protocol or just anywhere, twisted app either freezes or
> >> terminates, and this *only* happens when the app runs as a daemon,
> >> created using a service (.tac file).
> >> 
> >> I hope you can take a look at the class PolicyClient, why it causes
> >> problem to reactor main loop. Please point me to where the idea might
> >> come
> >> from.
> >> 
> >> I can give you the example but not sure if it is enough, as the client
> >> needs to connect to a server.
> > 
> > It looks like PolicyClient does a blocking HTTP request in __init__.
> > Since PolicyClient is instantiated in the reactor thread, this will block
> > the reactor.  Perhaps this call is hanging for some reason when the
> > process
> > has daemonized?  This would be unusual but not unheard of.  For example,
> > there are macOS environments where a process does not have access to the
> > network if it is not associated with a windowing session.
> > 
> > What platform do you observe the problem on, and what more can you learn
> > about what the process does on its way to hanging (eg can you run strace
> > on
> > it)?
> > 
> > Jean-Paul
> > 
> >> Thank you,
> >> TS
> >> 
> >> 
> >> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
> >> 
> >> exar...@twistedmatrix.com> wrote:
> >>> On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
> >>> 
> >>> touch.sereyse...@gmail.com> wrote:
>  Hello,
>  
>  I am developing a twisted app, and it runs as a service using twistd -y
>  to start the app.
>  I am having a problem of running a thread. I know it is not recommended
>  to use thread, but the library that I use, the object created is
>  running in
>  a thread.
>  
>  Here is the problem:
>  
>  1) if I start reactor by running reactor.run() directly, thread is
>  running fine
>  2) if I run it as a service using twisted, the thread is not running,
>  it runs but it seems to be blocked, because I tried to write something
>  to
>  file using time.sleep(), but file is empty.
>  
>  Is there something that I miss? How can I debug this?
> >>> 
> >>> A good next step would be to create at Short, Self Contained, Correct
> >>> (Compilable), Example  and share it.
> >>> 
> >>> Jean-Paul
> >>> 
>  Thank you,
>  
>  TS
>  ___
>  Twisted-Python mailing list
>  Twisted-Python@twistedmatrix.com
>  https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >>> 
> >>> ___
> >>> Twisted-Python mailing list
> >>> Twisted-Python@twistedmatrix.com
> >>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >> 
> >> ___
> >> Twisted-Python mailing list
> >> Twisted-Python@twistedmatrix.com
> >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > 
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted and Thread, thread not running when twisted app is running as service

2020-05-22 Thread Jean-Paul Calderone 
On Thu, May 21, 2020 at 4:46 PM Sereysethy TOUCH 
wrote:

> I am on MacOS, it is a development phase, but deployment will be on Linux.
>
> It is hard to run dtrace/dtruss on MacOS due to the "system integrity
> protection" on MacOS. I got some output but very limited information.
>
> So if there is a workaround you might suggest?
>

I would definitely recommend nailing down the exact cause of the problem.
I think that system integrity protection can be bypassed, perhaps by
creating a virtualenv?  I am not particularly familiar with macOS but I'm
sure there's some good information somewhere out there.

If the problem is the blocking network operations performed by
`PolicyClient` then you'll probably have to move even the instantiation of
that class to a separate thread.  This is probably good anyway, since it's
not necessarily the case that it is safe to create the object in one thread
and then use it in another.  Better to create and use it all in the same
thread.

Jean-Paul


>
> TS
>
> On Thu, May 21, 2020 at 10:20 PM Jean-Paul Calderone <
> exar...@twistedmatrix.com> wrote:
>
>> On Thu, May 21, 2020 at 3:56 PM Sereysethy TOUCH <
>> touch.sereyse...@gmail.com> wrote:
>>
>>> Hi Jean-Paul,
>>>
>>> I found the error after working on a short, self contained, correcte
>>> example. It is not about twisted and normal thread in general, but
>>> something else.
>>>
>>> After more tests, I come to realise that whenever I instantiate this
>>> class PolicyClient from this library
>>> https://github.com/ray-project/ray/blob/master/rllib/env/policy_client.py in
>>> my protocol or just anywhere, twisted app either freezes or terminates, and
>>> this *only* happens when the app runs as a daemon, created using a
>>> service (.tac file).
>>>
>>> I hope you can take a look at the class PolicyClient, why it causes
>>> problem to reactor main loop. Please point me to where the idea might come
>>> from.
>>>
>>> I can give you the example but not sure if it is enough, as the client
>>> needs to connect to a server.
>>>
>>
>> It looks like PolicyClient does a blocking HTTP request in __init__.
>> Since PolicyClient is instantiated in the reactor thread, this will block
>> the reactor.  Perhaps this call is hanging for some reason when the process
>> has daemonized?  This would be unusual but not unheard of.  For example,
>> there are macOS environments where a process does not have access to the
>> network if it is not associated with a windowing session.
>>
>> What platform do you observe the problem on, and what more can you learn
>> about what the process does on its way to hanging (eg can you run strace on
>> it)?
>>
>> Jean-Paul
>>
>>
>>>
>>> Thank you,
>>> TS
>>>
>>>
>>> On Thu, May 21, 2020 at 5:23 PM Jean-Paul Calderone <
>>> exar...@twistedmatrix.com> wrote:
>>>
 On Thu, May 21, 2020 at 10:58 AM Sereysethy TOUCH <
 touch.sereyse...@gmail.com> wrote:

> Hello,
>
> I am developing a twisted app, and it runs as a service using twistd
> -y to start the app.
> I am having a problem of running a thread. I know it is not
> recommended to use thread, but the library that I use, the object created
> is running in a thread.
>
> Here is the problem:
>
> 1) if I start reactor by running reactor.run() directly, thread is
> running fine
> 2) if I run it as a service using twisted, the thread is not running,
> it runs but it seems to be blocked, because I tried to write something to
> file using time.sleep(), but file is empty.
>
> Is there something that I miss? How can I debug this?
>

 A good next step would be to create at Short, Self Contained, Correct
 (Compilable), Example  and share it.

 Jean-Paul


>
> Thank you,
>
> TS
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
 ___
 Twisted-Python mailing list
 Twisted-Python@twistedmatrix.com
 https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python