[issue23382] Maybe can not shutdown ThreadPoolExecutor when call the method of shutdown
miles added the comment: The attachment includes the patch file -- keywords: +patch nosy: +milesli Added file: http://bugs.python.org/file38274/thread.py.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23382 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23382] Maybe can not shutdown ThreadPoolExecutor when call the method of shutdown
New submission from miles:
Maybe can not shutdown ThreadPoolExecutor when call the method shutdown.
Though the variable of _shutdown is set to true in the method of shutdown, it
may also reads the variable of _shutdown from cpu cache in the method of
_worker, and the worst case is that it could see an out-of-date value of
_shutdown forever. so need to acquire lock before reading the variable of
_shutdown to make sure see an up-to-date value.
the following is the new code:
def _worker(executor_reference, work_queue):
try:
while True:
work_item = work_queue.get(block=True)
if work_item is not None:
work_item.run()
continue
executor = executor_reference()
shutdown = False
with executor._shutdown_lock.acquire():
shutdown = executor._shutdown
# Exit if:
# - The interpreter is shutting down OR
# - The executor that owns the worker has been collected OR
# - The executor that owns the worker has been shutdown.
if _shutdown or executor is None or shutdown:
# Notice other workers
work_queue.put(None)
return
del executor
except BaseException:
_base.LOGGER.critical('Exception in worker', exc_info=True)
def shutdown(self, wait=True):
with self._shutdown_lock:
self._shutdown = True
self._work_queue.put(None)
if wait:
for t in self._threads:
t.join()
--
components: 2to3 (2.x to 3.x conversion tool)
messages: 235319
nosy: miles
priority: normal
severity: normal
status: open
title: Maybe can not shutdown ThreadPoolExecutor when call the method of
shutdown
versions: Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23382
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23382] Maybe can not shutdown ThreadPoolExecutor when call the method of shutdown
miles added the comment: the attachment includes the new code -- Added file: http://bugs.python.org/file38002/thread.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23382 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23382] Maybe can not shutdown ThreadPoolExecutor when call the method of shutdown
miles added the comment: The attachment includes the new code -- Added file: http://bugs.python.org/file37997/thread.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23382 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13405] Add DTrace probes
Changes by Chris Miles miles.ch...@gmail.com: -- nosy: -chrismiles ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13405 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles darryl.mi...@darrylmiles.org added the comment: With regards to create test cases for certain situations, sure this would be possible but not with pure python since your APIs deny/inhibit the particular things required to force a situation for a test case. With regards to SSL_peek() blocking, you'd need to explain yourself better on that one. The patch has been tested with the test cases from Python SVN enough to be happy they run ok. Maybe you have some offline not yet checked in SSL test cases you are referring to. To clarify why this is being done, if there is unread data then SSL_shutdown() will never return 1. Maybe you can simulate this situation by using SSL_write() with 1 byte payloads and a 10ms delay between each SSL_write() of the QUIT response message (you are trying to simulate network propagation delay). Then you have a client that tries to do unwrap() right after having sent the quit command, but makes no attempt to receive the response. I'll leave you guys too it about how you want to handle things with python (i.e. to make the design choice trade offs). I think all the points have been covered. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles darryl.mi...@darrylmiles.org added the comment: I am unable to get make test to run from an unpatched version in SVN (details below of make output). Please find attached an updated patch for your consideration (and testing, as I can't test it due to 'make test' SIGSEGV on CentOS 5.4 i386). Patch Notes: 1) Some thing that concern me, the unwrap() philosophy looks to be used to remove SSL from the Python high-level socket handle, so you can go back to plaintext mode. You can ONLY perform an unwrap() AFTER an SSL_shutdown()==1 has been observed (you need to wait for the other end to do something voluntarily). So you must retry the SSL_shutdown() over and over while you sleep-wait for IO, so this is akin to calling the ssl.shutdown(ssl.SSL_SHUTDOWN_MODE_BOTH) and getting back success. Also if it is your intention to properly implement an unwrap() like this you should disable IO read-ahead mode before calling shutdown for the second time, SSL_set_read_ahead(ssl, 0). This stops OpenSSL from eating too many bytes accidentally (probably from the kernel into its own buffers), from the inbound IO stream, which may not be SSL protocol data, it maybe plain text data (behind the last byte of SSL protocol data). 2) Due to the IO waiting it looks also necessary to copy the setup of SSL_set_nbio() from the read/write paths so the check_socket_and_wait_for_timeout() works in sympathy to the callers IO timeout reconfiguration. 3) My patch presumes the allocation of the type struct PySSLObject uses calloc() or some other memory zeroing strategy. There is a new member in that struct to track if SSL_shutdown() has previously returned a zero value. 4) The SSL_peek() error path needs checking to see if the error return is consistent with the Python paradigm. 5) Please check I have understand the VARARGS method correctly. I have made the default to SSL_SHUTDOWN_MODE_SENT (despite backward compatibly being SSL_SHUTDOWN_MODE_ONCE), this is because I would guess that most high-level applications did not intend to use it in raw mode; nor be bothered with the issues surrounding correct usage. I would guess high-level applications wanted Python to take the strain here. 6) I suspect you need to address your unwrap() policy a little better, the shutdown operation and the unwrap() are two different matters. The shutdown() should indicate success or not (in respect of the mode being requested, raw mode is a tricky one as the caller would want to the exact error return so it can do the correct thing), unwrap() should itself call ssl.shutdown(ssl.SSL_SHUTDOWN_MODE_BOTH) until it sees success and then remove the socket (and deallocate SSL objects). As things stand SSL_SHUTDOWN_MODE_ONCE does not work in a useful way since the error returns are not propagated to the caller, because unwrap is mixed into this. So that would still need fixing. building works ok, testing fails with SIGSEGV. Is this something to do with no having _bsddb built ? I have db-4.3 working. Maybe someone can reply by email on the matter. # make running build running build_ext building dbm using gdbm Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts # make test running build running build_ext building dbm using gdbm Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts find ./Lib -name '*.py[co]' -print | xargs rm -f ./python -Wd -3 -E -tt ./Lib/test/regrtest.py -l == CPython 2.7a4+ (trunk:79902M, Apr 11 2010, 16:38:55) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] == Linux-2.6.18-164.15.1.el5-i686-with-redhat-5.4-Final == /root/python-svn/build/test_python_29248 test_grammar test_opcodes test_dict test_builtin test_exceptions test_types test_unittest test_doctest test_doctest2 test_MimeWriter test_SimpleHTTPServer test_StringIO test___all__ /root/python-svn/Lib/test/test___all__.py:10: DeprecationWarning: in 3.x, the bsddb module has been removed; please use the pybsddb project instead import bsddb /root/python-svn/Lib/bsddb/__init__.py:67: PendingDeprecationWarning: The CObject type is marked Pending Deprecation in Python 2.7. Please use capsule objects instead. import _bsddb make: *** [test] Segmentation fault -- Added file: http://bugs.python.org/file16872/Modules__ssl.c.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles darryl.mi...@darrylmiles.org added the comment: To explain why you need 2 modes, a client/server would expect to do the following pseudo actions for maximum efficiency: set_socket_timeout(600_SECONDS) # or useful default send_data_over_ssl(QUIT\r\n) shutdown(SSL_SHUTDOWN_MODE_SENT) flush_data_down_to_socket() # maybe automatic/implied (OpenSSL users with custom BIO layers should be aware of this step) shutdown(socket, SHUT_WR) # this is optional, TCP socket level shutdown recv_data_over_ssl() = 250 Bye bye!\r\n # this will take time to arrive set_socket_io_timeout(5_SECONDS) shutdown(SSL_SHUTDOWN_MODE_BOTH) # this is optional! some clients may choose to skip it entirely close()/unwrap() A server would: recv_data_over_ssl() = QUIT\r\n # would be sitting idle waiting for this command send_data_over_ssl(250 Bye bye!\r\n) shutdown(SSL_SHUTDOWN_MODE_SENT) flush_data_down_to_socket() # maybe automatic/implied (OpenSSL users with custom BIO layers should be aware of this step) shutdown(socket, SHUT_WR) # this is optional, TCP socket level shutdown set_socket_io_timeout(30_SECONDS) shutdown(SSL_SHUTDOWN_MODE_BOTH) # a good server would implement this step close()/unwrap() Now if your outbound data is CORKed and flushed, the flush points would cause all the SSL data from both the 'last sent data' and the 'send shutdown notify' to go out in the same TCP segment and arrive at the other end more or less together. Doing any of the above in a different order introduces some kind of inefficiency. shutdown(fd, SHUT_WR) are often used at the socket level to help the manage TIME_WAIT. The client has to wait for the QUIT response message anyway. With the above sequence there is no additional time delay or cost with both parties performing a SSL protocol shutdown at the same time. Despite the IO timeouts existing (to provide a safety net). If the client is talking to a buggy server the worst case scenario is that it receives the quit response but the server never does an SSL shutdown and the server doesn't close the socket connection. In this situation the client will have to wait for IO timeout, some clients in other software use blocking sockets and don't have a timeout so they end up hooked (forever). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles darryl.mi...@darrylmiles.org added the comment: In order to build Python with a specific version of OpenSSL followed the CYGWIN instructions and edited Modules/Setup to make it read (note - I added -L$(SSL) into the linker options too, since by default on CentOS 5.4 i386 OpenSSL build in static library mode ala ../openssl-1.0.0/libssl.a) : SSL=../openssl-1.0.0 _ssl _ssl.c \ -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ -L$(SSL)/lib -L$(SSL) -lssl -lcrypto It is not clear to me what Python's goals are: * To be backward compatible, in which case I don't know your historical use of SSL_shutdown(). * To be a thin-layer (1:1) over OpenSSL, so that power users can harness the full potential of OpenSSL if they are willing to understand the finer points. * To provide a full-featured Python API. * To provide a Python API that is easy to use within the Python paradigm. These goals may not be convergent. -- nosy: +dlmiles Added file: http://bugs.python.org/file16838/python_ssl.c.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles darryl.mi...@darrylmiles.org added the comment: I've updated my attachment to the bug, if you read the old one please re-read the updated version (since some points in there were not accurate). With regards to the OpenSSL error return -1/ERROR_SYSCALL with errno==0 being observed, I shall respond into the OpenSSL mailing list with a fuller response. The man page SSL_get_error(3) does explain what getting a zero error means in relation to end-of-file at the BIO/TCP socket level. In light of the presumption by me that the problem was because one end did a syscall close(fd) this makes perfect sense in the context of your observation and OpenSSL appears to be working as documented. There is also code to print out the error in Python at Modules/_ssl.c method PySSL_SetError() so I'm not sure of the source of the funny looking error printing in relation to the ftpcli test case, consider it to be an error message formatting glitch. Now the issue I see here is that there are clearly 3 use cases Python should provide: * one-shot raw mode (don't enter the loop at all, as per newssl5.patch/my attachment, this is more or less what you already have in CVS, but I would remove the 2nd call to SSL_shutdown(), raw mode means exactly that; the caller is in charge of calling it again, thin layer for Python power users) [case-1] * perform IO sleep/wait as necessary until we observe SSL_shutdown()==0 (or better! so this will return if 0 or 1 is returned) [case-2] * perform IO sleep/wait as necessary until we observe SSL_shutdown()==1 [case-3] I presume you already have a way of handling the configuration of I/O timeouts as per Python's IO processing model (that is provided by some other API mechanism). The question is what is the best way to provide them (what is inline with the Python paradigm?) : * one method, keep existing named method, add new optional argument that can indicate all 3 modes of operation. Debate which of the 3 modes of operation is the default when the argument is not specified, case-1 seems to most backwardly compatible. [I am presuming python supports optional arguments] * new method, keep existing as-is (to cover case 1), implement case-2 and case-3 in the method which also take an argument for the user to specify which use case they want. From this a patch should be straight-forward. Then we can look to see if the FTP client or server is doing anything wrong in light of having the building blocks in place to achieve any goal on top of OpenSSL. -- Added file: http://bugs.python.org/file16845/python_ssl_v2.c.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Changes by Darryl Miles darryl.mi...@darrylmiles.org: Removed file: http://bugs.python.org/file16838/python_ssl.c.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8108 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2320] Race condition in subprocess using stdin
Changes by Chris Miles miles.ch...@gmail.com: -- nosy: +chrismiles ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2320 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5468] urlencode does not handle bytes, and could easily handle alternate encodings
Changes by Miles Kaufmann mile...@umich.edu: Removed file: http://bugs.python.org/file14796/urllib_parse.py3k.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5468] urlencode does not handle bytes, and could easily handle alternate encodings
Miles Kaufmann mile...@umich.edu added the comment: I've attached a patch that provides similar functionality to Dan Mahn's urlencode(), as well as providing encoding and errors parameters to parse_qs and parse_qsl, updating the documentation to reflect the added parameters, and adding test cases. The implementation of urlencode() is not the same as dmahn's, and has a more straightforward control flow and less code duplication than the current implementation. (For the tests, I tried to match the style of the file I was adding to with regard to (expect, result) order, which is why it's inconsistent.) -- keywords: +patch versions: +Python 3.2 Added file: http://bugs.python.org/file14796/urllib_parse.py3k.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6234] cgi.FieldStorage is broken when given POST data
Changes by Miles Kaufmann mile...@umich.edu: -- nosy: +milesck ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6234 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5468] urlencode does not handle bytes, and could easily handle alternate encodings
Miles Kaufmann mile...@umich.edu added the comment: parse_qs and parse_qsl should also grow encoding and errors parameters to pass to the underlying unquote(). -- nosy: +milesck ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4111] Add DTrace probes
Changes by Chris Miles miles.ch...@gmail.com: -- nosy: +chrismiles ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4111 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4736] io.BufferedRWPair.closed broken; tries to call bool writer.closed property
New submission from Miles semantic...@gmail.com:
The closed property of BufferedRWPair attempts to call the closed property
of its writer as a method, which fails because writer.closed is a bool.
The following code demonstrates the error:
import socket
socket.socket().makefile('rwb').closed
--
components: Library (Lib)
messages: 78257
nosy: semanticist
severity: normal
status: open
title: io.BufferedRWPair.closed broken; tries to call bool writer.closed
property
type: behavior
versions: Python 2.6, Python 3.0
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4736
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
