[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles added the comment: In order to build Python with a specific version of OpenSSL followed the CYGWIN instructions and edited Modules/Setup to make it read (note - I added "-L$(SSL)" into the linker options too, since by default on CentOS 5.4 i386 OpenSSL build in static library mode ala ../openssl-1.0.0/libssl.a) : SSL=../openssl-1.0.0 _ssl _ssl.c \ -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ -L$(SSL)/lib -L$(SSL) -lssl -lcrypto It is not clear to me what Python's goals are: * To be backward compatible, in which case I don't know your historical use of SSL_shutdown(). * To be a thin-layer (1:1) over OpenSSL, so that "power users" can harness the full potential of OpenSSL if they are willing to understand the finer points. * To provide a full-featured Python API. * To provide a Python API that is easy to use within the Python paradigm. These goals may not be convergent. -- nosy: +dlmiles Added file: http://bugs.python.org/file16838/python_ssl.c.txt ___ Python tracker <http://bugs.python.org/issue8108> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles added the comment: I've updated my attachment to the bug, if you read the old one please re-read the updated version (since some points in there were not accurate). With regards to the OpenSSL error return -1/ERROR_SYSCALL with errno==0 being observed, I shall respond into the OpenSSL mailing list with a fuller response. The man page SSL_get_error(3) does explain what getting a zero error means in relation to end-of-file at the BIO/TCP socket level. In light of the presumption by me that the problem was because one end did a syscall close(fd) this makes perfect sense in the context of your observation and OpenSSL appears to be working as documented. There is also code to print out the error in Python at Modules/_ssl.c method PySSL_SetError() so I'm not sure of the source of the funny looking error printing in relation to the ftpcli test case, consider it to be an error message formatting glitch. Now the issue I see here is that there are clearly 3 use cases Python should provide: * one-shot raw mode (don't enter the loop at all, as per newssl5.patch/my attachment, this is more or less what you already have in CVS, but I would remove the 2nd call to SSL_shutdown(), raw mode means exactly that; the caller is in charge of calling it again, thin layer for Python power users) [case-1] * perform IO sleep/wait as necessary until we observe SSL_shutdown()==0 (or better! so this will return if 0 or 1 is returned) [case-2] * perform IO sleep/wait as necessary until we observe SSL_shutdown()==1 [case-3] I presume you already have a way of handling the configuration of I/O timeouts as per Python's IO processing model (that is provided by some other API mechanism). The question is what is the best way to provide them (what is inline with the Python paradigm?) : * one method, keep existing named method, add new optional argument that can indicate all 3 modes of operation. Debate which of the 3 modes of operation is the default when the argument is not specified, case-1 seems to most backwardly compatible. [I am presuming python supports optional arguments] * new method, keep existing as-is (to cover case 1), implement case-2 and case-3 in the method which also take an argument for the user to specify which use case they want. >From this a patch should be straight-forward. Then we can look to see if the >FTP client or server is doing anything wrong in light of having the building >blocks in place to achieve any goal on top of OpenSSL. -- Added file: http://bugs.python.org/file16845/python_ssl_v2.c.txt ___ Python tracker <http://bugs.python.org/issue8108> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Changes by Darryl Miles : Removed file: http://bugs.python.org/file16838/python_ssl.c.txt ___ Python tracker <http://bugs.python.org/issue8108> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles added the comment: I am unable to get "make test" to run from an unpatched version in SVN (details below of make output). Please find attached an updated patch for your consideration (and testing, as I can't test it due to 'make test' SIGSEGV on CentOS 5.4 i386). Patch Notes: 1) Some thing that concern me, the "unwrap()" philosophy looks to be used to remove SSL from the Python high-level socket handle, so you can go back to plaintext mode. You can ONLY perform an unwrap() AFTER an SSL_shutdown()==1 has been observed (you need to wait for the other end to do something voluntarily). So you must retry the SSL_shutdown() over and over while you sleep-wait for IO, so this is akin to calling the ssl.shutdown(ssl.SSL_SHUTDOWN_MODE_BOTH) and getting back success. Also if it is your intention to properly implement an unwrap() like this you should disable IO read-ahead mode before calling shutdown for the second time, SSL_set_read_ahead(ssl, 0). This stops OpenSSL from eating too many bytes accidentally (probably from the kernel into its own buffers), from the inbound IO stream, which may not be SSL protocol data, it maybe plain text data (behind the last byte of SSL protocol data). 2) Due to the IO waiting it looks also necessary to copy the setup of SSL_set_nbio() from the read/write paths so the check_socket_and_wait_for_timeout() works in sympathy to the callers IO timeout reconfiguration. 3) My patch presumes the allocation of the type "struct PySSLObject" uses calloc() or some other memory zeroing strategy. There is a new member in that struct to track if SSL_shutdown() has previously returned a zero value. 4) The SSL_peek() error path needs checking to see if the error return is consistent with the Python paradigm. 5) Please check I have understand the VARARGS method correctly. I have made the default to SSL_SHUTDOWN_MODE_SENT (despite backward compatibly being SSL_SHUTDOWN_MODE_ONCE), this is because I would guess that most high-level applications did not intend to use it in raw mode; nor be bothered with the issues surrounding correct usage. I would guess high-level applications wanted Python to take the strain here. 6) I suspect you need to address your unwrap() policy a little better, the shutdown operation and the unwrap() are two different matters. The shutdown() should indicate success or not (in respect of the mode being requested, raw mode is a tricky one as the caller would want to the exact error return so it can do the correct thing), unwrap() should itself call ssl.shutdown(ssl.SSL_SHUTDOWN_MODE_BOTH) until it sees success and then remove the socket (and deallocate SSL objects). As things stand SSL_SHUTDOWN_MODE_ONCE does not work in a useful way since the error returns are not propagated to the caller, because "unwrap" is mixed into this. So that would still need fixing. building works ok, testing fails with SIGSEGV. Is this something to do with no having _bsddb built ? I have db-4.3 working. Maybe someone can reply by email on the matter. # make running build running build_ext building dbm using gdbm Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts # make test running build running build_ext building dbm using gdbm Python build finished, but the necessary bits to build these modules were not found: bsddb185 sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts find ./Lib -name '*.py[co]' -print | xargs rm -f ./python -Wd -3 -E -tt ./Lib/test/regrtest.py -l == CPython 2.7a4+ (trunk:79902M, Apr 11 2010, 16:38:55) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] == Linux-2.6.18-164.15.1.el5-i686-with-redhat-5.4-Final == /root/python-svn/build/test_python_29248 test_grammar test_opcodes test_dict test_builtin test_exceptions test_types test_unittest test_doctest test_doctest2 test_MimeWriter test_SimpleHTTPServer test_StringIO test___all__ /root/python-svn/Lib/test/test___all__.py:10: DeprecationWarning: in 3.x, the bsddb module has been removed; please use the pybsddb project instead import bsddb /root/python-svn/Lib/bsddb/__init__.py:67: PendingDeprecationWarning: The CObject type is marked Pending Deprecation in Python 2.7. Please use capsule objects instead. import _bsddb make: *** [test] Segmentation fault -- Added file: http://bugs.python.org/file16872/Modules__ssl.c.patch ___ Python tracker <http://bugs.python.org/issue8108> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles added the comment:
To explain why you need 2 modes, a client/server would expect to do the
following pseudo actions for maximum efficiency:
set_socket_timeout(600_SECONDS) # or useful default
send_data_over_ssl("QUIT\r\n")
shutdown(SSL_SHUTDOWN_MODE_SENT)
flush_data_down_to_socket() # maybe automatic/implied (OpenSSL users with
custom BIO layers should be aware of this step)
shutdown(socket, SHUT_WR) # this is optional, TCP socket level shutdown
recv_data_over_ssl() = "250 Bye bye!\r\n" # this will take time to arrive
set_socket_io_timeout(5_SECONDS)
shutdown(SSL_SHUTDOWN_MODE_BOTH) # this is optional! some clients may choose
to skip it entirely
close()/unwrap()
A server would:
recv_data_over_ssl() = "QUIT\r\n" # would be sitting idle waiting for this
command
send_data_over_ssl("250 Bye bye!\r\n")
shutdown(SSL_SHUTDOWN_MODE_SENT)
flush_data_down_to_socket() # maybe automatic/implied (OpenSSL users with
custom BIO layers should be aware of this step)
shutdown(socket, SHUT_WR) # this is optional, TCP socket level shutdown
set_socket_io_timeout(30_SECONDS)
shutdown(SSL_SHUTDOWN_MODE_BOTH) # a good server would implement this step
close()/unwrap()
Now if your outbound data is CORKed and flushed, the flush points would cause
all the SSL data from both the 'last sent data' and the 'send shutdown notify'
to go out in the same TCP segment and arrive at the other end more or less
together.
Doing any of the above in a different order introduces some kind of
inefficiency. shutdown(fd, SHUT_WR) are often used at the socket level to help
the manage TIME_WAIT.
The client has to wait for the QUIT response message anyway. With the above
sequence there is no additional time delay or cost with both parties performing
a SSL protocol shutdown at the same time. Despite the IO timeouts existing (to
provide a safety net).
If the client is talking to a buggy server the worst case scenario is that it
receives the quit response but the server never does an SSL shutdown and the
server doesn't close the socket connection. In this situation the client will
have to wait for IO timeout, some clients in other software use blocking
sockets and don't have a timeout so they end up hooked (forever).
--
___
Python tracker
<http://bugs.python.org/issue8108>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8108] test_ftplib fails with OpenSSL 0.9.8m
Darryl Miles added the comment: With regards to create test cases for certain situations, sure this would be possible but not with "pure python" since your APIs deny/inhibit the particular things required to force a situation for a test case. With regards to SSL_peek() blocking, you'd need to explain yourself better on that one. The patch has been tested with the test cases from Python SVN enough to be happy they run ok. Maybe you have some offline not yet checked in SSL test cases you are referring to. To clarify why this is being done, if there is unread data then SSL_shutdown() will never return 1. Maybe you can simulate this situation by using SSL_write() with 1 byte payloads and a 10ms delay between each SSL_write() of the "QUIT response message" (you are trying to simulate network propagation delay). Then you have a client that tries to do unwrap() right after having sent the quit command, but makes no attempt to receive the response. I'll leave you guys too it about how you want to handle things with python (i.e. to make the design choice trade offs). I think all the points have been covered. -- ___ Python tracker <http://bugs.python.org/issue8108> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
