Checking network input processing by Python for a multi-threaded server
Hello, I constructed another multi-threaded TCP server for my needs (based on the available software documentation). https://docs.python.org/3/library/socketserver.html#asynchronous-mixins I constructed also a corresponding script which should send nine record sets (which get extracted from a simple JSON file) to this server on my local host by the software “Python 3.7.2-3.1” (for an openSUSE system). Related background information can be seen for a discussion topic like “Data exchange over network interfaces by SmPL scripts”. https://systeme.lip6.fr/pipermail/cocci/2019-April/005792.html https://lore.kernel.org/cocci/6ec5b70f-39c3-79e5-608f-446a870f0...@web.de/ The file name for the client script is passed by a parameter to a command which is repeated by this server in a loop. It is evaluated then how often a known record set count was sent. I got a test result like the following. elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 list_last_two_statements_from_if_branches-statistic-server3.py "return code"|"received records"|"command output"|incidence 0|9||63 0|5||5 0|13||2 0|10||5 0|11||7 0|8||3 0|7||3 0|14||3 0|6||4 0|12||3 0|4||2 real1m23,301s user0m6,434s sys 0m1,430s * How should this data processing approach be adjusted so that the same number of records will be handled in a consistent way? * Which challenges from inter-process communication should be taken better into account for the involved programming interfaces (because of multi-threading)? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking support for efficient appending to mapped data
> The file name for the client script is passed by a parameter to a command > which is repeated by this server in a loop. Additional explanations can be helpful for the shown software situation. > It is evaluated then how often a known record set count was sent. It was actually determined for a specific function variant how many input record sets were available for data processing by the server in a loop iteration. > I got a test result like the following. > > > elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 > list_last_two_statements_from_if_branches-statistic-server3.py > "return code"|"received records"|"command output"|incidence > 0|9||63 > 0|5||5 > 0|13||2 … Such a test result shows that some records could be handled by the server within the time range from the start of a command and the exit of the corresponding child process which sends desired data back to its caller. A fraction of records can be handled only at a subsequent iteration if no session management was applied as in this test case. > * How should this data processing approach be adjusted so that the same number > of records will be handled in a consistent way? There is a need to add session management so that required context information will be connected with the provided input data. Each session needs to be identified by an unique key. The Python programming language provides the data type “dict” for such a mapping as standard functionality. https://docs.python.org/3/library/stdtypes.html#mapping-types-dict Python's support for associative containers is generally nice. But I find that consequences from modification of mapped data can become more interesting for the mentioned use case. The operator “+=” (augmented assignment) can be used there, can't it? The appending of data is supported for some types by this operation. I am informed in the way that some data types expect that a well-known method like “append” must (or should be) be called instead for such an “addition” (if you care for efficient data processing). https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types Now I am looking again for further clarifications and improvements in this area. Would you like to share any more ideas here? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> Does this mean that the second number in a listing like the above > should be approximatively equal? The expectation (or hope) was that an identical number of record sets would be available for each loop iteration in such a data processing approach by the server. But the reality is different for the observed input value distribution. > Then, obviously, your threads must communicate among themselves > and wait in case they have too much run ahead. This is a software design option. I am looking again for adjustments around programming interfaces for efficient appending to mapped data according to required context management. > As you say to have created a "multi-threaded TCP server", I assume > that your server spawns a thread for each TCP connection. Is this the software behaviour we usually get from the class “socketserver.ThreadingMixIn”? > Then, you can concentrate on proper multi-thread synchronization > and forget about "inter-process communication challenges". I showed an experiment where I put a loop around a test command. I am wondering also about varying data processing results from the execution of a single command. The visualisation of output statistics seems to be more challenging for such a test case. Can any other test tools help a bit more here? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> * Which challenges from inter-process communication should be taken better > into > account for the involved programming interfaces (because of > multi-threading)? I would like to clarify another recurring challenge with network configuration. A port number should be selected somehow for the service where the desired input data will be received. It can be nice when a fixed port can be specified for such data exchange. Advanced service management can achieve a reasonably safe number allocation. But I would occasionally prefer a dynamic selection for some data processing approaches on my test systems. How does the support look like for the handling of ephemeral ports by programming interfaces for Python? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> In Python, there's a certain amount of support. You can attempt to > bind to a port, and if you fail, try the next one in a sequence. * The zero seems to be also an usable parameter here. Is the system configuration documentation unclear about the applied value range for the port allocation? Can it happen that special permissions would be needed for a setting? * How are the chances to improve programming interfaces around the module “socketserver”? > It's simple, it's straight-forward, and it doesn't cause problems. Would you like to reduce connection management difficulties because of an error message like “OSError: [Errno 98] Address already in use”? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> You'll get anything in the ephemeral ports range. From which documentation source did you get this information for the handling of a zero as an useful setting? > But the other cause is that you recently shut the server down, > and the port is still in the TIME_WAIT state. Does this technical detail occasionally increase the need to choose an additional number for a quickly restarted service in a dynamic way? > You can avoid this with the SO_REUSEADDR flag. Can such a configuration parameter be used also together with programming interfaces from the module “socketserver”? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> If your server listens on a random port how does the client know > which port to connect to? I am fiddling also with the data processing variant that such connection properties are passed as parameters for a command which is executed as a child process so that desired data can be sent back to its caller. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> https://docs.python.org/3/library/socketserver.html#asynchronous-mixins An example is provided also in this software documentation. May I expect that data were completely received from clients and accordingly processed by the request handler in the started threads after the statement “server.shutdown()” was sucessfully executed? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> https://docs.python.org/3/library/socketserver.html#asynchronous-mixins I have constructed a pair of small scripts for another test. A command (which refers to the second Python script) is executed 100 times by “subprocess.run()” with parameters so that the child process can send six test records back to the caller over a TCP connection. 1. The received records are appended to a global list variable during each loop iteration. 2. The list length is appended to another global list variable. 3. The stored list lengths are counted by grouping of this information. Now I wonder again about a test result like the following for the software “Python 3.7.2-3.1” (for an openSUSE system). elfring@Sonne:~/Projekte/Python> time /usr/bin/python3 test-statistic-server1.py incidence|"available records"|"return code"|"command output" 44|6|0| 12|5|0| 13|4|0| 16|3|0| 2|7|0| 8|2|0| 3|8|0| 1|1|0| 1|9|0| real0m29,123s user0m5,925s sys 0m1,073s Does this data processing approach indicate a need for further software corrections? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> Why not provide them so that anyone trying to analyze what you are > seeing can try them for themselves. I assume that an other information system can be more appropriate for file distribution than this mailing list. > Starting a full process takes time This is usual. - Such an action would be necessary for the test case. > Do you have ANY synchronization enabled, I would expect that this is provided by the run time system for the Python scripting language. > so that you start all the children The main process waits on the exit for the single started command in each loop iteration. > and they all wait for a single "go" signal to be set No. > -- so that they all start processing at the same time? I suggest to take another look at the description for the test algorithm. > No idea what is in each loop... An instance for the class “threaded_TCP_server” and a call of “subprocess.run(…)”. > If there is only one list (implied by #1) There are two lists adjusted. 1. Storage for received records before the statement “server.shutdown()” (and a reset to an empty list) is executed at the end of each loop iteration. 2. List lengths are recorded for analysis after the data collection phase. > Again, it is not clear what is being counted here... How many record sets were stored and are accessible at one point in time in such a data structure? > What is an "incidence", how is "available records" determined? The generated table shows how often a specific number of record sets was temporarily available for further data processing. I would expect that the correct count should be 100 (once) here (instead of the display of deviations from the original six test records). > Also, TCP is a stream protocol, not a record protocol. This view is generally fine. JSON data structures are transmitted by the discussed approach. > A client could use, say, 6 separate .send() operations, This should be happening in this test. > while the server receives everything in a single .recv() call. I hope that the execution of the statement “server.shutdown()” triggers a separation in the reported inter-process communication. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> https://docs.python.org/3.4/library/socketserver.html > about the middle of the page... It seems that you would like to refer to an other document. https://docs.python.org/3/library/socket.html#socket-example Under which circumstances will the execution of the method “socket.setsockopt” be triggered by a class like “socketserver.TCPServer”? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Dynamic selection for network service ports?
> For the truly lazy, we have hash links. Thanks for your reminder. > https://docs.python.org/3/library/socketserver.html#socketserver.BaseServer.allow_reuse_address The relationship of this class attribute with the identifier (or option) “SO_REUSEADDR” might become easier to find. The use of such a configuration parameter might need further considerations. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
>> May I expect that data were completely received from clients and accordingly >> processed by the request handler in the started threads after >> the statement “server.shutdown()” was sucessfully executed? > > Python delegates those low level services to the underlaying > network library, which likely will implement the TCP specification. * How do you think about data processing consequences when record sets are concurrently received and are appended to a global list (for a Python variable)? * Would you expect that the run time system takes care for data consistency because of involved multi-threading? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
>> An instance for the class “threaded_TCP_server” and a call of >> “subprocess.run(…)”. >> > And how many connections does the subprocess make? A new TCP connection is performed with the information from the passed parameters for each call of the function “sendall”. The test command will send JSON data over these six connections then. > A full connect(), send(), close() for each output? Yes, for this test variant. > OTOH, if you are making multiple connect() calls, then OS scheduling > could result in multiple server threads running. This functionality is desired. … > Of course, you still have to use your head! … > For instance, it makes no sense to use a forking server … I would expect that the main test process will not be forked here. >> I hope that the execution of the statement “server.shutdown()” triggers >> a separation in the reported inter-process communication. > > .shutdown() stops the server from processing new connections... I hope so, too. (Additional server objects can be created in subsequent loop iterations.) How does this interpretation fit to the wording “Tell the serve_forever() loop to stop and wait until it does.” from the documentation of the class “socketserver.BaseServer”? > It does nothing for the nature of TCP streams This aspect should be fine. > I'd be tempted to convert the master and subprocess from TCP to UDP, > just to see if there is a difference. I do not want to change the data transmission technique for this use case. I suggest to take another look at a related discussion topic like “Data exchange over network interfaces by SmPL scripts” if you would prefer to review a concrete source code example instead of recognising data processing consequences from a known test algorithm. https://systeme.lip6.fr/pipermail/cocci/2019-April/005792.html https://lore.kernel.org/cocci/6ec5b70f-39c3-79e5-608f-446a870f0...@web.de/ Will it help to add a bug report in an issue tracker? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> In any multi-threaded application, you must be carefull when > accessing (and especially modifying) shared (e.g. "global") objects. > In general, you will need locks to synchronize the access. I agree to this general view. > Appending to a list (and some other elementary operations > on Python data structures) is protected by Python's "GIL" > (= "Global Interpreter Lock") and thereby becomes "atomic". > Thus, if all you do is appending - then the append has no need > of explicite locking. Thanks for such information. > It does not protect the integrity of your data structures. It can be that the data protection needs to be extended occasionally. But I am more interested in the detail that a specific Python list variable should reflect the received record sets from a single test command in a consistent way. Did all extra worker threads exit after the statement “server.shutdown()” was successfully executed? > Thus, if your threads modify shared objects, then you are responsible > to protect access to them with appropriate locking. How do you think about to improve the distinction for the really desired lock granularity in my use case? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> I suggested UDP as a TEST, not for the end use... I can understand such a suggestion. Can it distract from other software surprises? > If UDP gives you the results you expect, it most likely means there is a > problem There is a questionable software behaviour still waiting for a proper solution (without switching the data transmission technique for this use case). > in how you are processing TCP data. Which impressions did you get from the implementation of the published Python functions “receive_data” and “receive_message” (according to the previously mentioned links)? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> Server.shutdown() sets a flag that tells the main server to /stop > accepting new requests/. Can it be that this method should perform a bit more resource management (according to the selected configuration like “socketserver.ThreadingMixIn”)? > So far as I can tell, for a threaded server, any threads/requests that > were started and haven't completed their handlers will run to completion -- > however long that handler takes to finish the request. Whether > threaded/forked/single-thread -- once a request has been accepted, it will > run to completion. This is good to know and such functionality fits also to my expectations for the software behaviour. > Executing .shutdown() will not kill unfinished handler threads. I got a result like the following from another test variant on a Linux system. elfring@Sonne:~/Projekte/Python> time python3 test-statistic-server2.py incidence|"available records"|"running threads"|"return code"|"command output" 80|6|1|0| 1|4|3|0| 1|4|4|0| 3|5|2|0| 1|5|3|0| 5|7|1|0| 1|3|4|0| 1|8|1|0| 1|4|1|0| 3|5|1|0| 1|4|2|0| 1|3|2|0| 1|6|2|0| real0m48,373s user0m6,682s sys 0m1,337s >> How do you think about to improve the distinction for the really >> desired lock granularity in my use case? > > If your request handler updates ANY shared data, YOU have to code the > needed MUTEX locks into that handler. I suggest to increase the precision for such a software requirement. > You may also need to code logic to ensure any handler threads have completed Can a class like “BaseServer” be responsible for the determination if all extra started threads (or background processes) finished their work as expected? > before your main thread accesses the shared data I became unsure at which point a specific Python list variable will reflect the received record sets from a single test command in a consistent way according to the discussed data processing. > -- that may require a different type of lock; I am curious on corresponding software adjustments. > something that allows multiple threads to hold in parallel > (unfortunately, Event() and Condition() aren't directly suitable) Which data and process management approaches will be needed finally? > Condition() with a global counter (the counter needs its own lock) > might work: handler does something like How much do the programming interfaces from the available classes support the determination that submitted tasks were completely finished? > The may still be a race condition on the last request if it is started > between the .shutdown call and the counter test (ie; the main submits > .shutdown, server starts a thread which doesn't get scheduled yet, > main does the .acquire()s, finds counter is 0 so assumes everything is done, > and THEN the last thread gets scheduled and increments the counter. Should mentioned system constraints be provided already by the Python function (or class) library? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> If you have a multi-threaded application and you want to be on > the "safe side", you always use your own locks. I suggest to reconsider your software expectations around the word “always”. There are more software design options available. > Python uses locks to protect its own data structures. > Whether this protection is enough for your use of Python types > depends on details you may not want to worry about. I agree to such a general view. > For example: most operations on Python types are atomic > (if they do not involve some kind of "waiting" or "slow operation") > *BUT* if they can detroy objects, then arbitrary code > (from destructors) can be executed and then they are not atomic. The safe handling of finalizers can trigger development challenges. > As an example "list.append" is atomic (no object is detroyed), > but "list[:] = ..." is not: while the list operation itself > is not interrupted by another thread, the operation may destroy > objects (the old list components) and other threads may get control > before the assignment has finished. How would you determine (with the help of the Python function/class library) that previously submitted tasks were successfully executed? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
>>> Server.shutdown() sets a flag that tells the main server to /stop >>> accepting new requests/. >> >> Can it be that this method should perform a bit more resource management >> (according to the selected configuration like “socketserver.ThreadingMixIn”)? >> > There isn't much more it can do I see further software design possibilities. > -- it has been a long standing axiom that killing threads is not recommended. This data processing approach will trigger various development challenges. >>> You may also need to code logic to ensure any handler threads have completed >> >> Can a class like “BaseServer” be responsible for the determination >> if all extra started threads (or background processes) finished their work >> as expected? > > You are asking for changes to the Python library for a use case > that is not common. I find this view questionable. > Normally connections to a server are independent and do not share common data Would you like to clarify corresponding application statistics any further? > -- if there is anything in common, Do you identify any more shared functionality? > it is likely stored in a database management system An use case evolved into my need to work also with an ordinary Python list variable for a simple storage interface. > which itself will provide locking for updates, It is nice when you can reuse such software. > and the connection handler will have to be coded to handle retries > if multiple connections try to update the same records. I am not concerned about this aspect for my test case. > Servers aren't meant to be started and shutdown at a rapid rate Their run times can vary considerably. > (it's called "serve_forever" for a reason). Will an other term become more appropriate? > If the socketserver module doesn't provide what you need, It took a while to understand the observed software behaviour better. > you are free to copy socketserver.py to some other file (myserver.py?), > and modify it to fit your needs. Will it help to clarify any software extensions with corresponding maintainers? > Maybe have the function that spawns handler threads append the thread ID > to a list, have the function that cleans up a handler thread at the end > send its ID via a Queue object, This approach can be reasonable to some degree. > and have the master periodically (probably the same loop that checks > for shutdown), read the Queue, and remove the received ID from the list > of active threads. I imagine that there are nicer design options available for notifications according to thread terminations. > On shutdown, you loop reading the Queue and removing IDs from the list > of active threads until the list is empty. Will a condition variable (or a semaphore) be more helpful here? >> How much do the programming interfaces from the available classes support >> the determination that submitted tasks were completely finished? >> > Read the library reference manual and, for those modules with Python > source, the source files (threading.py, socketserver.py, queue.py...) > > The simpler answer is that these modules DON'T... I suggest to improve the software situation a bit more. > It is your responsibility. This view can be partly appropriate. > socketserver threading model is that the main server loops waiting > for connection requests, when it receives a request it creates > a handler thread, and then it completely forgets about the thread I find that this technical detail can be better documented. > -- the thread is independent and completes the handling of the request. Would you occasionally like to wait on the return value from such data processing? (Are these threads joinable?) > If you need to ensure everything has finished before starting > the next server instance, you will have to write the extensions > to socketserver. I might achieve something myself while it can be also nice to achieve adjustments together with other developers. >> Should mentioned system constraints be provided already by the Python >> function (or class) library? > > Again, the model for socketserver, especially in threaded mode, is that > requests being handled are completely independent. shutdown() merely stops > the master from responding to new requests. I find this understanding of the software situation also useful. > In a more normal situation, .shutdown() would be called > and then the entire program would call exit. * Do you expect any more functionality here than an exit from a single thread? * Does this wording include the abortion of threads which were left over? > It is NOT normal for a program to create a server, shut it down, > only to then repeat the sequence. Will your understanding of such an use case grow, too? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> And in addition, you can derive your own class from `socketserver` > and override methods to provide whatever additional functionality > you think is necessary. Such a programming possibility remains generally. I am curious if the software development attention can be adjusted also in this area. I imagine that it can be easier to depend on a higher level system infrastructure. How do you think about to reuse software extensions around the standard “CORBA” like “omniORB”? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> Nowadays, I develop typically web applications.
> There, something similar to "CORBA" is used: WSDL described
> "web services". Typically, they use the web infrastructure
> and its protocols ("http", "https") for communication.
The popularity varies for these programming interfaces over time.
* I am trying to get also a data processing variant working
based on the JSON format together with an extended socket
server class.
* Do you eventually know any proxy services which would provide
useful data type conversions?
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> Nowadays, I develop typically web applications.
> There, something similar to "CORBA" is used: WSDL described
> "web services". Typically, they use the web infrastructure
> and its protocols ("http", "https") for communication.
The popularity varies for these programming interfaces over time.
* I am trying to get also a data processing variant working
based on the JSON format together with an extended socket
server class.
* Do you eventually know any proxy services which would provide
useful data type conversions?
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> socketserver threading model is that the main server loops waiting for > connection requests, when it receives a request it creates a handler thread, This data processing style can be generally fine as long as you would like to work without a thread (or process) pool. > and then it completely forgets about the thread I have taken another look at the implementation of the corresponding methods. https://github.com/python/cpython/blob/3.7/Lib/socketserver.py#L656 I get an other impression from the statements “self._threads.append(t)” (process_request) and “thread.join()” (server_close). > -- the thread is independent and completes the handling of the request. Will a corresponding return value bet set? > If you need to ensure everything has finished before starting the next server > instance, I am still looking for the support of software constraints in this direction. > you will have to write the extensions to socketserver. Will the development situation evolve any more here? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
>> I get an other impression from the statements “self._threads.append(t)” >> (process_request) >> and “thread.join()” (server_close). > > Okay -- v3.7 has added more logic that didn't exist in the v3.5 code > I was examining... (block_on_close is new). Thanks for such a version comparison. > However, I need to point out that this logic is part of server_close(), > which is not the same as shutdown(). You have been calling shutdown() which > only ends the loop accepting new connections, but leaves any in-process > threads to finish handling their requests. > > server_close() needs to be called by your code, I would expect AFTER > calling shutdown() to stop accepting new requests (and starting new threads > which may not be in the list that the close is trying to join). Should this aspect be taken into account by the code specification “with server:”? > And after calling server_close() you will not be able to simply "restart" the > server > -- you must go through the entire server initialization process > (ie: create a whole new server instance). This should be finally achieved by the implementation of my method “perform_command”. I hope that corresponding data processing can be cleanly repeated then as desired for test purposes. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking network input processing by Python for a multi-threaded server
> The file name for the client script is passed by a parameter to a command > which is repeated by this server in a loop. > It is evaluated then how often a known record set count was sent. In which time ranges would you expect the receiving of the complete JSON data which were sent by the child process (on the local host during my test)? Can the program termination be taken into account for waiting on still incoming data from the server socket? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Checking refusal of a network connection
Hello,
I can start a service as desired.
elfring@Sonne:~/Projekte/Bau/C++/test-statistic-server1/local>
./test-statistic-server2 & /usr/bin/ss -t -l -p -H|grep test
[1] 8961
waiting for connections
server_id: localhost
server_port: 35529
LISTEN 0 123 [::1]:35529 [::]:*
users:(("test-statistic-",pid=8961,fd=3))
elfring@Sonne:~/Projekte/Bau/C++/test-statistic-server1/local> 0 connections
were handled.
But I wonder about the following error message then.
elfring@Sonne:~/Projekte/Python> /usr/bin/python3
~/Projekte/Python/socket-send_json_data.py --server_id localhost --server_port
35529
Using Python version:
3.7.2 …
Traceback …:
…
File "/home/elfring/Projekte/Python/socket-send_json_data.py", line 17, in
send_data
so.connect((args.server_id, args.server_port))
ConnectionRefusedError: [Errno 111] Connection refused
How should this inter-process communication difficulty be resolved?
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
> Well, providing minimal code samples that produce the problem would be
> a start.
I prefer an other approach to clarify relevant software configuration
differences.
> Otherwise we are just guessing...
I can offer other data before.
> Maybe you have a firewall problem.
I hope not.
I can try another server variant out as expected.
elfring@Sonne:~/Projekte/Python> /usr/bin/python3 test-server2.py &
[1] 14067
elfring@Sonne:~/Projekte/Python> /usr/bin/ss -t -l -p -H|grep python
LISTEN0 5 127.0.0.1:search-agent 0.0.0.0:*
users:(("python3",pid=14067,fd=3))
elfring@Sonne:~/Projekte/Python> /usr/bin/python3 socket-send_json_data.py
--server_id localhost --server_port 1234
Using Python version:
3.7.2 (default, Dec 30 2018, 16:18:15) [GCC]
elfring@Sonne:~/Projekte/Python> Result:
…
Can connections work also with a network service address like “[::1]:35529”
(which would be used by the C++ server implementation so far)?
How does the software situation look like for the support of the IPv6 loopback
address?
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
> Also, it can be very useful to strace the client process, eg:
Do you find the following background information more helpful
for the desired clarification of unexpected software behaviour?
elfring@Sonne:~/Projekte/Python> LANG=C strace -e trace=network
/usr/bin/python3 socket-send_json_data.py --server_id localhost --server_port
37351
Using Python version:
3.7.2 …
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 5
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(4, "\2\0\0\0\r\0\0\0\6\0\0\0hosts\0", 18, MSG_NOSIGNAL, NULL, 0) = 18
recvmsg(4, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="hosts\0",
iov_len=6}, {iov_base="\310O\3\0\0\0\0\0", iov_len=8}], msg_iovlen=2,
msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS,
cmsg_data=[5]}], msg_controllen=20, msg_flags=MSG_CMSG_CLOEXEC},
MSG_CMSG_CLOEXEC) = 14
connect(3, {sa_family=AF_INET, sin_port=htons(37351),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
Traceback …:
…
File "socket-send_json_data.py", line 17, in send_data
so.connect((args.server_id, args.server_port))
ConnectionRefusedError: [Errno 111] Connection refused
+++ exited with 1 +++
> You can also strace the running service process:
I do not observe additional function calls for the TCP client connection
attempt here.
> Also, on the service side it isn't enough to create the service socket,
> you also need to do an accept IIRC.
This should be performed by my implementation of the C++ function “setup”.
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
>> connect(3, {sa_family=AF_INET, sin_port=htons(37351),
>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
>
> Without seeing the code, I'd be suspicious of that difference.
I would expect that the IPv4 address from such a connection attempt
would be automatically converted to a IPv6 loopback address.
Unfortunately, the direct specification “… socket-send_json_data.py --server_id
::1 …”
does not work at the moment because of the error message “socket.gaierror:
[Errno -9]
Address family for hostname not supported”.
Regards,
Markus
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
>> I would expect that the IPv4 address from such a connection attempt >> would be automatically converted to a IPv6 loopback address. > > You haven't said which OS you are using, but as far as I know this > expectation will be frustrated at least on Linux: There ::1 and > 127.0.0.1 are distinct addresses. How does this view fit to information from the Linux programmer's manual? See also: command “man 7 ipv6” Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
> Which specific information in that man page contradicts what I wrote? We can agree that the mentioned IP addresses are distinct. But the corresponding functionality should be equivalent. > If you think of > > | IPv4 connections can be handled with the v6 API by using the > | v4-mapped-on-v6 address type; thus a program needs to support only > | this API type to support both protocols. > > please note that 127.0.0.1 mapped to IPv6 is ::7f00:1, not ::1. I find another information like “This is handled transparently by the address handling functions in the C library.” also interesting. > So you still need to bind to two addresses. I am unsure about this conclusion. Under which circumstances will the Python programming interfaces support the direct usage of the identification “::1”? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
> It looks like the service isn't listening at the time the so.connect is > called. * I get an other impression from the programs “/usr/bin/netstat” and “/usr/bin/ss”. * The data transmission seems to work also for my small script “socket-send_test_data1.tcl” (even when the identification “::1” was passed as a command parameter). Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking support for network connections from Python client to IPv6 server
> "Handled transparently" means that an ipv6 server can handle connections > from ipv4 clients without doing anything special. It is nice if this conversion is working. > They just appear to come from a specific IPv6 address range. I got expected connections by my small script “socket-send_test_data1.tcl”. >> Under which circumstances will the Python programming interfaces >> support the direct usage of the identification “::1”? > > I'm not sure I understand the question. They do. How would like to explain the error message “socket.gaierror: [Errno -9] Address family for hostname not supported” on my Linux test system then? Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking support for network connections from Python client to IPv6 server
>> How would like to explain the error message “socket.gaierror: [Errno -9] >> Address family for hostname not supported” on my Linux test system then? > > Can you supply a tiny standalone piece of code demonstrating this error > please? The following script part would be relevant. … def send_data(x): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as so: global args so.connect((args.server_id, args.server_port)) … If the address family “AF_INET6” was passed instead, the identification “::1” can work also as a command parameter. The data transmission seems to succeed by my script “socket-send_json_data2.py” then. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking support for network connections from Python client to IPv6 server
>> How would like to explain the error message “socket.gaierror: [Errno -9] >> Address family for hostname not supported” on my Linux test system then? > > Can you supply a tiny standalone piece of code demonstrating this error > please? The following script part would be relevant. … def send_data(x): with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as so: global args so.connect((args.server_id, args.server_port)) … If the address family “AF_INET6” was passed instead, the identification “::1” can work also as a command parameter. The data transmission seems to succeed by my script “socket-send_json_data2.py” then. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
Re: Checking refusal of a network connection
> How would this conversion take place? Localhost is 127.0.0.1. > Localhost6 is ::1. They are different My configuration file “/etc/hosts” provides the following information as usual. “… ::1 localhost ipv6-localhost ipv6-loopback …” > and you cannot route between the two. I got other expectations for the corresponding software behaviour. > What I can see is that your server binds to localhost6 and your client > is trying to connect to localhost. I am curious to clarify the circumstances further if such a combination can also work finally. If my software test client would pass the IPv6 address family for a connection, both processes would use the same network protocol version. Regards, Markus -- https://mail.python.org/mailman/listinfo/python-list
