Re: How to shut down a TCPServer serve_forever() loop?

[Ian Kelly]

On Tue, Nov 28, 2017 at 11:54 AM, Marko Rauhamaa  wrote:
> Chris Angelico :
>> Would the OP have been trivially able to send a signal to the
>> process? Yes.
>
>Python signal handlers are always executed in the main Python thread,
>even if the signal was received in another thread. This means that
>signals can’t be used as a means of inter-thread communication.
>
>https://docs.python.org/3/library/signal.html#signals-and-threads>

Being received in the other thread, it would still interrupt the
system call however, wouldn't it? Quoting from the doc:

"However, if the target thread is executing the Python interpreter,
the Python signal handlers will be executed by the main thread.
Therefore, the only point of sending a signal to a particular Python
thread would be to force a running system call to fail with
InterruptedError."
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Chris Angelico :
> On Wed, Nov 29, 2017 at 5:32 AM, Marko Rauhamaa  wrote:
>> Don't you worry about my programs.
>
> Okay, but you can't claim that problems are solvable if you cheat them.

What I'm saying is that there's no particular reason why glibc couldn't
offer a solution. There *is* getaddrinfo_a(), but that's suboptimal
because it uses signals and (probably) a subsidiary thread. Instead, it
should offer a file descriptor for the application to monitor.

>> Please reread the original poster's question. It was about a blocking
>> TCP listener call that another thread couldn't interrupt.
>
> Yet a SIGINT would successfully interrupt it.

A keyboard interrupt? That your magic bullet? How does that work in
practice?

> Would the OP have been trivially able to send a signal to the
> process? Yes.

   Python signal handlers are always executed in the main Python thread,
   even if the signal was received in another thread. This means that
   signals can’t be used as a means of inter-thread communication.

   https://docs.python.org/3/library/signal.html#signals-and-threads>


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Chris Angelico]

On Wed, Nov 29, 2017 at 5:32 AM, Marko Rauhamaa  wrote:
> Chris Angelico :
>
>> On Wed, Nov 29, 2017 at 5:03 AM, Marko Rauhamaa  wrote:
>>> Chris Angelico :
 Do you respect /etc/nsswitch.conf?
>>>
>>> No, but I don't need to.
>>
>> Ah, right. Until the day you're wrestling with "why doesn't /etc/hosts
>> apply to this program". Yep, you totally don't need nsswitch.
>
> Don't you worry about my programs.

Okay, but you can't claim that problems are solvable if you cheat them.

>>> In this discussion I was referring to the fact that you can interrupt
>>> a coroutine while that is generally not possible to do to a blocking
>>> thread.
>>
>> I'm not sure what you mean by a "blocking thread". Whether it's a
>> coroutine or not, you can't interrupt gethostbyname(); and whether
>> it's a coroutine or not, you CAN interrupt any syscall that responds
>> to signals (that's the whole point of EINTR).
>
> Please reread the original poster's question. It was about a blocking
> TCP listener call that another thread couldn't interrupt.

Yet a SIGINT would successfully interrupt it. I'm not sure what your
point is. Would the OP have been trivially able to switch to asyncio?
Maybe. Would the OP have been trivially able to send a signal to the
process? Yes.

I'm done arguing. You're clearly not listening.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Chris Angelico :

> On Wed, Nov 29, 2017 at 5:03 AM, Marko Rauhamaa  wrote:
>> Chris Angelico :
>>> Do you respect /etc/nsswitch.conf?
>>
>> No, but I don't need to.
>
> Ah, right. Until the day you're wrestling with "why doesn't /etc/hosts
> apply to this program". Yep, you totally don't need nsswitch.

Don't you worry about my programs.

>> In this discussion I was referring to the fact that you can interrupt
>> a coroutine while that is generally not possible to do to a blocking
>> thread.
>
> I'm not sure what you mean by a "blocking thread". Whether it's a
> coroutine or not, you can't interrupt gethostbyname(); and whether
> it's a coroutine or not, you CAN interrupt any syscall that responds
> to signals (that's the whole point of EINTR).

Please reread the original poster's question. It was about a blocking
TCP listener call that another thread couldn't interrupt.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Chris Angelico]

On Wed, Nov 29, 2017 at 5:03 AM, Marko Rauhamaa  wrote:
> Chris Angelico :
>
>> On Wed, Nov 29, 2017 at 4:22 AM, Marko Rauhamaa  wrote:
>>> I have solved the gethostbyname() problem by implementing the DNS
>>> protocol myself (in Python).
>>
>> Do you respect /etc/nsswitch.conf?
>
> No, but I don't need to.

Ah, right. Until the day you're wrestling with "why doesn't /etc/hosts
apply to this program". Yep, you totally don't need nsswitch.

 I don't understand why you keep insisting that asyncio and threads
 are somehow incompatible,
>>>
>>> Where did I say that?
>>
>> You talk about them as if they're fundamentally different.
>
> In this discussion I was referring to the fact that you can interrupt a
> coroutine while that is generally not possible to do to a blocking
> thread.

I'm not sure what you mean by a "blocking thread". Whether it's a
coroutine or not, you can't interrupt gethostbyname(); and whether
it's a coroutine or not, you CAN interrupt any syscall that responds
to signals (that's the whole point of EINTR).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Chris Angelico :

> On Wed, Nov 29, 2017 at 4:22 AM, Marko Rauhamaa  wrote:
>> I have solved the gethostbyname() problem by implementing the DNS
>> protocol myself (in Python).
>
> Do you respect /etc/nsswitch.conf?

No, but I don't need to.

>>> I don't understand why you keep insisting that asyncio and threads
>>> are somehow incompatible,
>>
>> Where did I say that?
>
> You talk about them as if they're fundamentally different.

In this discussion I was referring to the fact that you can interrupt a
coroutine while that is generally not possible to do to a blocking
thread.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Chris Angelico]

On Wed, Nov 29, 2017 at 4:22 AM, Marko Rauhamaa  wrote:
> I have solved the gethostbyname() problem by implementing the DNS
> protocol myself (in Python).

Do you respect /etc/nsswitch.conf?

>> I don't understand why you keep insisting that asyncio and threads are
>> somehow incompatible,
>
> Where did I say that?

You talk about them as if they're fundamentally different.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Chris Angelico :

> On Tue, Nov 28, 2017 at 11:52 PM, Marko Rauhamaa  wrote:
>> The original poster's problem seems to be caused by blocking APIs that
>> cannot be multiplexed using select(). A good many Python facilities are
>> the same way.
>>
>> Such blocknoxious APIs are at the core of the multithreading programming
>> paradigm.
>
> Some things are fundamentally not multiplexable, at the lower levels.

Said function is not. Neither are, say, database methods or web
operations. Yet, they are commonly implemented in a blocking fashion.

> (On Linux, and I believe most other Unix-like operating systems,
> there's no non-blocking way to open a file, nor to gethostbyname.)

The file access problem is a fundamental glitch in Linux, but
gethostbyname() is just a silly POSIX API.

> How do you propose to solve those in Python?

I have solved the gethostbyname() problem by implementing the DNS
protocol myself (in Python).

The file access issue is philosophically somewhat deep; Linux
essentially treats files as memory and vice versa. If it became an
issue, I'd have to launch a file server process.

> I don't understand why you keep insisting that asyncio and threads are
> somehow incompatible,

Where did I say that?


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Chris Angelico]

On Tue, Nov 28, 2017 at 11:52 PM, Marko Rauhamaa  wrote:
> Chris Angelico :
>
>> On Tue, Nov 28, 2017 at 5:04 PM, Marko Rauhamaa  wrote:
>>> Seems to be one of the fundamental multithreading issues: each thread
>>> is blocked on precisely one event. Asyncio is more flexible: you can
>>> multiplex on a number of events.
>>
>> Not really, no. Unless select() counts as "precisely one event", of
>> course.
>
> Select() counts as asyncio.

Hmm. So what DOESN'T count as asyncio? Because I can use
multithreading with select, and in fact have done so on many
occasions. It's a perfectly normal Unix kernel function.

>> That's the normal way for a thread to block on multiple events on a
>> Unix system. Windows has its own approximate equivalent.
>>
>> Surprise, surprise, that's also how event loops often implemented.
>> Including ones used in packages like asyncio.
>
> The original poster's problem seems to be caused by blocking APIs that
> cannot be multiplexed using select(). A good many Python facilities are
> the same way.
>
> Such blocknoxious APIs are at the core of the multithreading programming
> paradigm.

Some things are fundamentally not multiplexable, at the lower levels.
(On Linux, and I believe most other Unix-like operating systems,
there's no non-blocking way to open a file, nor to gethostbyname.) How
do you propose to solve those in Python? Do you have one thread that
uses select() and another that does a blocking call? Do you spin off a
thread to do the blocking call and then have that thread notify your
main thread via a file descriptor?

I don't understand why you keep insisting that asyncio and threads are
somehow incompatible, or that they're a dichotomy. They're not. They
work together very nicely.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Chris Angelico :

> On Tue, Nov 28, 2017 at 5:04 PM, Marko Rauhamaa  wrote:
>> Seems to be one of the fundamental multithreading issues: each thread
>> is blocked on precisely one event. Asyncio is more flexible: you can
>> multiplex on a number of events.
>
> Not really, no. Unless select() counts as "precisely one event", of
> course.

Select() counts as asyncio.

> That's the normal way for a thread to block on multiple events on a
> Unix system. Windows has its own approximate equivalent.
>
> Surprise, surprise, that's also how event loops often implemented.
> Including ones used in packages like asyncio.

The original poster's problem seems to be caused by blocking APIs that
cannot be multiplexed using select(). A good many Python facilities are
the same way.

Such blocknoxious APIs are at the core of the multithreading programming
paradigm.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Chris Angelico]

On Tue, Nov 28, 2017 at 5:04 PM, Marko Rauhamaa  wrote:
> Ian Kelly :
>
>> On Sat, Nov 25, 2017 at 7:10 AM, John Pote  
>> wrote:
>>> The issue is that if I press a key on the keyboard the key is
>>> immediately shown on the screen but then the shutdown() call blocks
>>> until another TCP connection is made, text is echoed back and only
>>> then does serve_forever()return followed by shutdown()returning as
>>> can be seen from the console session,
>> [...]
>> Make a connection to the server after calling shutdown to wake up the
>> server's event loop? I'm guessing it only checks the shutdown flag
>> after responding to an event, so there's probably not much else you
>> could do.
>
> Seems to be one of the fundamental multithreading issues: each thread is
> blocked on precisely one event. Asyncio is more flexible: you can
> multiplex on a number of events.

Not really, no. Unless select() counts as "precisely one event", of course.

http://man7.org/linux/man-pages/man2/select.2.html
https://en.wikipedia.org/wiki/Select_(Unix)

That's the normal way for a thread to block on multiple events on a
Unix system. Windows has its own approximate equivalent.

Surprise, surprise, that's also how event loops often implemented.
Including ones used in packages like asyncio.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Marko Rauhamaa]

Ian Kelly :

> On Sat, Nov 25, 2017 at 7:10 AM, John Pote  wrote:
>> The issue is that if I press a key on the keyboard the key is
>> immediately shown on the screen but then the shutdown() call blocks
>> until another TCP connection is made, text is echoed back and only
>> then does serve_forever()return followed by shutdown()returning as
>> can be seen from the console session,
> [...]
> Make a connection to the server after calling shutdown to wake up the
> server's event loop? I'm guessing it only checks the shutdown flag
> after responding to an event, so there's probably not much else you
> could do.

Seems to be one of the fundamental multithreading issues: each thread is
blocked on precisely one event. Asyncio is more flexible: you can
multiplex on a number of events.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to shut down a TCPServer serve_forever() loop?

[Ian Kelly]

On Sat, Nov 25, 2017 at 7:10 AM, John Pote  wrote:
> Hi all,
>
> My problem in summary is that my use of the shutdown() method only shuts
> down a server after the next TCP request is received.
>
> I have a TCP server created in the run() method of a thread.
>
> class TCPlistener( Thread ):
> def run( self ):
>with socketserver.TCPServer(  ("localhost", ), ConnHandler )
> as server:
> self.server = server
> print( "TCP listener on: %s:%d" % ( self.host, self.port ) )
> self.server.serve_forever()
>
> print( "TCPlistener:run() ending" )
>
> ConnHandler is a simple echo class with only a handle() method
>
> The main bit of the program is
>
> if __name__ == "__main__":
> serverThrd = TCPlistener()
> serverThrd.start()#start TCP IP server listening
> print("server started")
>
> ch = getche()#wait for key press
> print()
> serverThrd.server.shutdown()
>
> print( "main ending" )
>
> Everying works as expected, numerous connections can be made and the
> received text is echoed back.
>
> The issue is that if I press a key on the keyboard the key is immediately
> shown on the screen but then the shutdown() call blocks until another TCP
> connection is made, text is echoed back and only then does
> serve_forever()return followed by shutdown()returning as can be seen from
> the console session,
>
>>>python36 TCPIPserver.py
> server started
> TCP listener on: localhost:
> q #pressed 'q' key
> 127.0.0.1 wrote: #Sent some text from PuTTY
> b'SOME TEXT'
> TCPlistener:run() ending
> main ending
>
> How can I get shutdown()to shut down the server immediately without waiting
> for the next TCP connection?

Make a connection to the server after calling shutdown to wake up the
server's event loop? I'm guessing it only checks the shutdown flag
after responding to an event, so there's probably not much else you could do.
-- 
https://mail.python.org/mailman/listinfo/python-list