[issue13103] copy of an asyncore dispatcher causes infinite recursion

[Xavier de Gaye]

New submission from Xavier de Gaye :

A regression occurs in python 3.2 when doing a copy of an asyncore
dispatcher.

$ python3.1
Python 3.1.2 (r312:79147, Apr  4 2010, 17:46:48) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore, copy
>>> copy.copy(asyncore.dispatcher())



$ python3.2
Python 3.2 (r32:88445, Jun 18 2011, 20:30:18) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore, copy
>>> copy.copy(asyncore.dispatcher())
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.2/copy.py", line 97, in copy
return _reconstruct(x, rv, 0)
  File "/usr/local/lib/python3.2/copy.py", line 291, in _reconstruct
if hasattr(y, '__setstate__'):
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
retattr = getattr(self.socket, attr)
  
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
retattr = getattr(self.socket, attr)
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
retattr = getattr(self.socket, attr)
RuntimeError: maximum recursion depth exceeded while calling a Python object


This occurs after the 'copy' module has created the new instance with
__new__(). This new instance does not have the 'socket' attribute,
hence the infinite recursion.

Adding the following methods to the dispatcher class, fixes the infinite
recursion:

def __getstate__(self):
state = self.__dict__.copy()
return state

def __setstate__(self, state):
self.__dict__.update(state)

But it does not explain why the recursion occurs in 3.2 and not in
3.1.

--
components: Extension Modules
messages: 144925
nosy: xdegaye
priority: normal
severity: normal
status: open
title: copy of an asyncore dispatcher causes infinite recursion
type: behavior
versions: Python 3.2

___
Python tracker 
<http://bugs.python.org/issue13103>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13103] copy of an asyncore dispatcher causes infinite recursion

[Xavier de Gaye]

Xavier de Gaye  added the comment:

The infinite recursion occurs also when running python 3.2 with the
extension modules copy, copyreg and asyncore from python 3.1. So it
seems this regression is not caused by a modification in these
modules.

Anyway, the bug is in asyncore. The attached patch fixes it and is
more robust than adding the __getstate__ and __setstate__ methods to
dispatcher.

The patch includes a test case.

--
keywords: +patch
Added file: http://bugs.python.org/file23317/infinite_recursion_asyncore.patch

___
Python tracker 
<http://bugs.python.org/issue13103>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13103] copy of an asyncore dispatcher causes infinite recursion

[Xavier de Gaye]

Xavier de Gaye  added the comment:

About why the asyncore bug shows up in python 3.2:

The simple test below is ok with python 3.1 but triggers a
"RuntimeError: maximum recursion depth exceeded..." with python 3.2:

$ python3.1
Python 3.1.2 (r312:79147, Apr  4 2010, 17:46:48) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...   def __getattr__(self, attr):
... return getattr(self.foo, attr)
... 
>>> c = C()
>>> hasattr(c, 'bar')
False
>>> 

For the reasoning behind this change made in python 3.2, see issue
9666 and the mail
http://mail.python.org/pipermail/python-dev/2010-August/103178.html

--

___
Python tracker 
<http://bugs.python.org/issue13103>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13103] copy of an asyncore dispatcher causes infinite recursion

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> So, in 3.1 hasattr(y, '__setstate__') *did* recurse and hit the limit,
> but the exception was caught and hasattr returned False?

This is right.


> I think I prefer the new behavior...
> The patch looks good, I would simply have raised AttributeError(name)
> though.

It is fine with me to raise AttributeError(name).
Note that when raising AttributeError('socket'), the user gets
notified of the exceptions on both 'socket' and 'name'.
For example with the patch applied:

$ python3
Python 3.2 (r32:88445, Jun 18 2011, 20:30:18) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore
>>> a = asyncore.dispatcher()
>>> del a.socket
>>> a.foo
Traceback (most recent call last):
  File "asyncore.py", line 415, in __getattr__
retattr = getattr(self.socket, attr)
  File "asyncore.py", line 413, in __getattr__
% self.__class__.__name__)
AttributeError: dispatcher instance has no attribute 'socket'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1, in 
  File "asyncore.py", line 418, in __getattr__
%(self.__class__.__name__, attr))
AttributeError: dispatcher instance has no attribute 'foo'

--

___
Python tracker 
<http://bugs.python.org/issue13103>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13183] pdb skips frames after hitting a breakpoint and running step

[Xavier de Gaye]

New submission from Xavier de Gaye :

Pdb skips frames after hitting a breakpoint and running step commands
that walk back the frame stack.

Run the following two tests with the two files named foo.py and bar.py:

===   foo.py   
from bar import bar

def foo():
bar()

def nope():
pass

def foobar():
foo()
nope()

foobar()

===   bar.py   
def bar():
print('1')

===   test_1   
$ python3 --version
Python 3.2

$ python3 -m pdb foo.py
> /path/to/foo.py(1)()
-> from bar import bar
(Pdb) from bar import bar
(Pdb) break bar
Breakpoint 1 at /path/to/bar.py:1
(Pdb) continue
> /path/to/bar.py(2)bar()
-> print('1')
(Pdb) step
1
--Return--
> /path/to/bar.py(2)bar()->None
-> print('1')
(Pdb) step
--Call--
> /path/to/foo.py(6)nope()
-> def nope():
(Pdb)

===   test_2   
$ python3 -m pdb foo.py
> /path/to/foo.py(1)()
-> from bar import bar
(Pdb) break nope
Breakpoint 1 at /path/to/foo.py:6
(Pdb) from bar import bar
(Pdb) break bar
Breakpoint 2 at /path/to/bar.py:1
(Pdb) continue
> /path/to/bar.py(2)bar()
-> print('1')
(Pdb) step
1
--Return--
> /path/to/bar.py(2)bar()->None
-> print('1')
(Pdb) step
--Return--
> /path/to/foo.py(4)foo()->None
-> bar()
(Pdb)

===

Note: stop_here, break_anywhere and dispatch_call are methods of the
Bdb class.

test_1 fails to stop in foo() after the second 'step' command because
the trace function is not set for all the frames being created in the
foo module, since stop_here() and break_anywhere() are both False
whenever dispatch_call() is invoked in this module. So after the
second 'step' command, trace_dispatch is not invoked by the
interpreter until a new frame is created, which happens when nope() is
called.

test_2 succeeds and stops in foo() after the second 'step' command.
After setting the dummy breakpoint 1 in the foo module in test_2,
break_anywhere() becomes True in the foo module and the trace function
is set for all the frames created in this module (with an associated
performance penalty).

The problem exists in all python versions.

The attached patch fixes this problem by restoring the trace function
on returning from a frame when the command is 'step'.

The patch includes a test case.

--
components: Library (Lib)
files: restore_trace.patch
keywords: patch
messages: 145557
nosy: xdegaye
priority: normal
severity: normal
status: open
title: pdb skips frames after hitting a breakpoint and running step
versions: Python 3.2
Added file: http://bugs.python.org/file23410/restore_trace.patch

___
Python tracker 
<http://bugs.python.org/issue13183>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13183] pdb skips frames after hitting a breakpoint and running step

[Xavier de Gaye]

Changes by Xavier de Gaye :


--
type:  -> behavior

___
Python tracker 
<http://bugs.python.org/issue13183>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13183] pdb skips frames after hitting a breakpoint and running step

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Uploaded restore_trace_2.patch that improves the test case.

--
Added file: http://bugs.python.org/file23418/restore_trace_2.patch

___
Python tracker 
<http://bugs.python.org/issue13183>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13183] pdb skips frames after hitting a breakpoint and running step

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Uploaded restore_trace.py27.patch with a fix and test case for python 2.7.

--
Added file: http://bugs.python.org/file23419/restore_trace.py27.patch

___
Python tracker 
<http://bugs.python.org/issue13183>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5661] asyncore should catch EPIPE while sending() and receiving()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> I am not marking 'test needed' since the problem is 'hardly reproducible'.

The attached script named 'asyncore_epipe.py' may be used to reproduce
systematically the EPIPE error on linux with Python 3.2:

* the Reader closes the socket after receiving the first data
* the Writer gets an exception when attempting to write the next
  message on the closed socket

When you run 'python asyncore_epipe.py' or 'python asyncore_epipe.py
1025' you get an EPIPE error (1025 is the size of exchanged messages,
so the last Writer message has been fully read by the Reader before
closing).

When you run 'python asyncore_epipe.py 128' you get an ECONNRESET (the
Reader reads only 128 bytes before closing the socket)

Note that ECONNRESET has been removed in this script, from the
frozenset of handled errors, to make this last point visible.

So it seems that, on linux, when writing to a closed socket, you get
an ECONNRESET when there is still data in the socket, and an EPIPE
otherwise. In the first case the tcp connection ends with a single
RESET, and in the second case it ends with the sequence FIN-ACK-RESET.

--
nosy: +xdegaye
Added file: http://bugs.python.org/file23528/asyncore_epipe.py

___
Python tracker 
<http://bugs.python.org/issue5661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5661] asyncore should catch EPIPE while sending() and receiving()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Uploaded the patch 'epipe-default.patch' with a test case that breaks
on linux when EPIPE is not handled by asyncore, which is the case with
Python 3.2 and previous versions.

--
Added file: http://bugs.python.org/file23533/epipe-default.patch

___
Python tracker 
<http://bugs.python.org/issue5661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5661] asyncore should catch EPIPE while sending() and receiving()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Uploaded the same test case for Python 2.7.

--
Added file: http://bugs.python.org/file23534/epipe-2.7.patch

___
Python tracker 
<http://bugs.python.org/issue5661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5661] asyncore should catch EPIPE while sending() and receiving()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> Did you try with the current branches?

Yes, the test is pass against the current default and 2.7 branches.
One must remove EPIPE from the asyncore._DISCONNECTED frozenset to
make the test to fail.


> Yes, see RFC1122 section 4.2.2.13:
> """
> A host MAY implement a "half-duplex" TCP close sequence, so
> that an application that has called CLOSE cannot continue to
> read data from the connection.  If such a host issues a
> CLOSE call while received data is still pending in TCP, or
> if new data is received after CLOSE is called, its TCP
> SHOULD send a RST to show that data was lost.
> """

Thanks for the pointer. Note that the "half-duplex" FIN-ACK-RESET
occurs when all received data has been read, and a single RESET is
sent when received data is still pending in the stream.

--

___
Python tracker 
<http://bugs.python.org/issue5661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5661] asyncore should catch EPIPE while sending() and receiving()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

The test fails when use_poll is True.
The difference between using poll() and poll2():

poll: All the read events are processed before the write events,
so the close after the first recv by TestHandler will be followed
by a send by TestClient within the same call to poll(). The
test is deterministic.

poll2: The order in which events are received is os dependent. So
it is possible that the first recv by TestHandler is the last
event in the 'r' list, so that after the close, a new call to
poll2() is done and the first event in this new 'r' list, is not
the expected write event for TestClient.

What about forcing self.use_poll to False, before calling
loop_waiting_for_flag() ? The drawback being that the test will be run
twice with the same environment.

--

___
Python tracker 
<http://bugs.python.org/issue5661>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> Actually the class asyncore.dispatcher_with_send do not handle
> properly disconnection. When the endpoint shutdown his sending part
> of the socket, but keep the socket open in reading, the current
> implementation of dispatcher_with_send will close the socket without
> sending pending data.
>
> Adding this method to the class fix the problem:
>
> def handle_close(self):
> if not self.writable():
> dispatcher.close()


This does not seem to work. The attached Python 3.2 script
'asyncore_12498.py' goes into an infinite loop of calls to
handle_read/handle_close when handle_close is defined as above. In
this script the Writer sends a 4096 bytes message when connected, the
Reader reads only 64 bytes and closes the connection afterwards. Then
follows the sequence:

select() returns a read event handled by handle_read()
handle_read() calls recv()
socket.recv() returns 0 to indicate a closed connection
recv() calls handle_close

This sequence is repeated forever in asyncore.loop() since out_buffer
is never empty.

Note that after the first 'handle_close' has been printed, there are
no 'handle_write' printed, which indicates that the operating system
(here linux) states that the socket is not writable.

--
nosy: +xdegaye
Added file: http://bugs.python.org/file23547/asyncore_12498.py

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Thanks for the explanations.
I confirm that the patch fixes 'asyncore_12498.py' with your changes
applied to this script.

Note that the patch may break applications that have given different
semantics to 'closing' ('closing' being such a common name for a
network application) after they noticed that this attribute is never
used by asyncore nor by asynchat.

It seems that the same fix could also be applied to asynchat.

--

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> Note that the patch may break applications that have given different
> semantics to 'closing' ('closing' being such a common name for a
> network application) after they noticed that this attribute is never
> used by asyncore nor by asynchat.

I was thinking about the discussion in issue 1641, the second part of
the discussion starting with msg 84972.

The attached patch uses another name and drains the output buffer only
on a close event, not on error conditions.

I will do the patch for asynchat and do both test cases, unless you
beat me to it.

--
Added file: http://bugs.python.org/file23555/asyncore_drain_2.diff

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Changes by Xavier de Gaye :


Added file: http://bugs.python.org/file23559/asyncore_shutdown.py

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

While writing the test case, I found out that the test case does not
fail before the patch. It seems that draining the output buffer
already works:

The attached script 'asyncore_shutdown.py' drains the output buffer
when run without the patch, with Python 3.2, and prints 'Done.'. The
dispatcher_with_send handle_close() method is never called.

The attached 'asyncore_shutdown.log' file is the output of the tcpdump
of the connection. It shows that packets are sent after the first FIN.
This is on linux.

--
Added file: http://bugs.python.org/file23560/asyncore_shutdown.log

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> That's because you didn't define a handle_read() method

Ooops...

Attached is a patch for dispatcher_with_send and asynchat with a test
case for each, that fail when the fix is not applied.

--
Added file: http://bugs.python.org/file23563/asyncore_drain_3.diff

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13310] asyncore handling of out-of-band data fails

[Xavier de Gaye]

New submission from Xavier de Gaye :

Add the following lines to test_handle_expt (this makes sense, a
dispatcher instance is supposed to implement handle_read and call recv
in order to detect that the remote end has closed the socket):

--- a/Lib/test/test_asyncore.py
+++ b/Lib/test/test_asyncore.py
@@ -677,6 +677,9 @@
 def handle_expt(self):
 self.flag = True
 
+def handle_read(self):
+self.recv(1)
+
 class TestHandler(BaseTestHandler):
 def __init__(self, conn):
 BaseTestHandler.__init__(self, conn)

With these lines added, the test now fails on linux with Python 3.3,
see the following backtrace: select (an poll) returns a read event and
an exceptional condition for the socket, but there is no normal data
to read, only out-of-band data.

The attached patch fixes the problem.


==
ERROR: test_handle_expt (test.test_asyncore.TestAPI_UseIPv4Poll)
--
Traceback (most recent call last):
  File "/path_to/src/cpython/cpython-hg-default/Lib/test/test_asyncore.py", 
line 690, in test_handle_expt
self.loop_waiting_for_flag(client)
  File "/path_to/src/cpython/cpython-hg-default/Lib/test/test_asyncore.py", 
line 523, in loop_waiting_for_flag
asyncore.loop(timeout=0.01, count=1, use_poll=self.use_poll)
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 215, in 
loop
poll_fun(timeout, map)
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 196, in 
poll2
readwrite(obj, flags)
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 117, in 
readwrite
obj.handle_error()
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 108, in 
readwrite
obj.handle_read_event()
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 439, in 
handle_read_event
self.handle_read()
  File "/path_to/src/cpython/cpython-hg-default/Lib/test/test_asyncore.py", 
line 681, in handle_read
self.recv(1)
  File "/path_to/src/cpython/cpython-hg-default/Lib/asyncore.py", line 379, in 
recv
data = self.socket.recv(buffer_size)
BlockingIOError: [Errno 11] Resource temporarily unavailable

--
components: Library (Lib)
files: handle_expt.diff
keywords: patch
messages: 146783
nosy: xdegaye
priority: normal
severity: normal
status: open
title: asyncore handling of out-of-band data fails
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file23580/handle_expt.diff

___
Python tracker 
<http://bugs.python.org/issue13310>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13311] asyncore handle_read should call recv

[Xavier de Gaye]

New submission from Xavier de Gaye :

When the remote end disconnects, handle_close is only called if recv
is called (from handle_read). The default implementation of
handle_read does not call recv.

Not having the default implementation of handle_read call recv, has
the following drawbacks:

an implementation of a subclass of dispatcher that only sends
data, a logger for example, may believe that it does not have to
implement handle_read since it does not expect any data and since
there is no hint in the code or in the documentation that it
should

test_handle_expt currently succeeds when it should fail since the
current handling of out-of-band data is broken (see issue 13310),
but if the default implementation of handle_read had called recv,
then test_handle_expt would have failed, allowing to detect the
problem

The attached patch adds a call to recv in handle_read, updates the
documentation and adds a test case.

Note that when this patch is applied, test_handle_expt fails
as expected, issue 13310 should be fixed first.

--
components: Library (Lib)
files: handle_read.diff
keywords: patch
messages: 146785
nosy: xdegaye
priority: normal
severity: normal
status: open
title: asyncore handle_read should call recv
type: behavior
versions: Python 2.7, Python 3.3
Added file: http://bugs.python.org/file23581/handle_read.diff

___
Python tracker 
<http://bugs.python.org/issue13311>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13310] asyncore handling of out-of-band data fails

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Hi Charles-François,

> And indeed, that's a known kernel regression introduced in 2.6.28,
> and fixed by this commit:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b634f87522dff87712df8bda2a6c9061954d552a
> http://kerneltrap.org/mailarchive/linux-netdev/2010/3/15/6271951

The BlockingIOError exception occurs on linux 2.6.30 for me.


> Note that there might still be a problem with the current code:
> recv() can return EAGAIN on a FD reported readable/writable by
> select() (for example if the network stack received an input packet
> and then discards it because of an invalid checksum, or because the
> output socket buffer has room left but not enough to accomodate the
> packet we're trying to send): I'll have to think a bit to see if we
> can do something about this, but that's another issue.

It is not clear why recv() can return EAGAIN because we're trying to
_send_ a packet ;-)


> Closing as invalid

Ok, 2.6.30 is an old kernel and urgent data is mostly never used.

--

___
Python tracker 
<http://bugs.python.org/issue13310>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13325] no address in the representation of asyncore dispatcher after connection established

[Xavier de Gaye]

New submission from Xavier de Gaye :

When an asyncore dispatcher initiates a tcp connection, its
representation lacks the peer address. The output of the attached
script 'dispatcher_addr.py' gives on linux with Python 3.2:

call stack in handle_connect_event:
->main->loop->poll->write->handle_write_event->handle_connect_event
self.addr after connection: None


The attached patch fixes the problem. The patch includes a test case.
The following comment in the patch:

# EWOULDBLOCK may also be returned by winsock when calling connect
# while the connection attempt is in progress

refers to the following statement in
http://msdn.microsoft.com/en-us/library/aa923167.aspx

"""
As a result, it is not recommended that applications use multiple
calls to connect to detect connection completion. If they do, they
must be prepared to handle WSAEINVAL and WSAEWOULDBLOCK error codes
the same way that they handle WSAEALREADY to ensure robust execution.
"""


Note that two consecutive calls to
socket.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) are made when
handle_write_event() is called and self.connected is False: one in
handle_write_event and the following right away in
handle_connect_event. This seems useless.

--
components: Library (Lib)
messages: 146849
nosy: xdegaye
priority: normal
severity: normal
status: open
title: no address in the representation of asyncore dispatcher after connection 
established
type: behavior
versions: Python 3.2

___
Python tracker 
<http://bugs.python.org/issue13325>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13325] no address in the representation of asyncore dispatcher after connection established

[Xavier de Gaye]

Changes by Xavier de Gaye :


Added file: http://bugs.python.org/file23593/dispatcher_addr.py

___
Python tracker 
<http://bugs.python.org/issue13325>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13325] no address in the representation of asyncore dispatcher after connection established

[Xavier de Gaye]

Changes by Xavier de Gaye :


--
keywords: +patch
Added file: http://bugs.python.org/file23594/dispatcher_addr.diff

___
Python tracker 
<http://bugs.python.org/issue13325>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Review done after Charles-François review.

--

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Attached is a new patch following the code review.

After rewriting the asyncore test to check that the data has been
successfully received by the client, the test fails when
using poll() and a data size of 4096 bytes. The reason is that when
TestHandler closes the connection after writing the output buffer, the
client receives a POLLHUP which prevents it to receive the data since
POLLHUP is triggering a call to handle_close.

POSIX states:
"""
POLLHUP
A device has been disconnected, or a pipe or FIFO has been closed
by the last process that had it open for writing. Once set, the
hangup state of a FIFO shall persist until some process opens the
FIFO for writing or until all read-only file descriptors for the
FIFO are closed. This event and POLLOUT are mutually-exclusive; a
stream can never be writable if a hangup has occurred. However,
this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not
mutually-exclusive. This flag is only valid in the revents
bitmask; it shall be ignored in the events member.
"""

The attached patch changes the handling of POLLHUP to fix this: it
calls a new method named handle_hangup_evt() on a POLLHUP event, and
uses a new _hangup attribute to have writable() return False after a
POLLHUP. Note that we do get a close event (on linux) when all data
has been received, allowing the client to close the socket.

Please review this new patch.

--
Added file: http://bugs.python.org/file23604/half_duplex_close.diff

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13311] asyncore handle_read should call recv

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> I'd say your patch can be useful only in case the dispatcher subclass
> doesn't send() neither recv() any data, in which case the connection
> is supposed to remain open forever.

There are some cases where it is important to detect that the remote
end is disconnected even if there is no data to send. Say a logger
connected to a data collector that sends data every few minutes. The
data collector dies, the logger may have to take actions on this
event: connect to a backup collector, raise an alarm, whatever... It
should not have to depend on the fact that data needs to be sent to
learn of the disconnection.

> Perhaps you could provide more info about why you needed to do this
> in the first place.

See also issue 12498 and the message 146653. When the remote end
performs a half-duplex disconnection, you may send data without
detecting the close event in send(), so you must rely on recv() to
detect it.

--

___
Python tracker 
<http://bugs.python.org/issue13311>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13311] asyncore handle_read should call recv

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> In this kind of situation, it is perfectly legitimate for the client
> to perform a half-duplex close (shutdown(SHUT_WR)), since it does
> not intend to send data (which is implied by the fact that the sever
> doesn't implement an handle_read() method).

> With the current code, the server will keep sending data until the
> remote end close()s its socket.
> With this patch, this would break: the server's handle_close()
> method will be called right away.

Since this patch may break existing valid code, I think it should be
closed as invalid.

--

___
Python tracker 
<http://bugs.python.org/issue13311>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> I think the best would be to not handle POLLHUP events while POLLIN
> is set, so that the handlers can have a chance to drain the input
> socket buffer.

Ok.


> But it's a separate issue, could you create a new one?

The test case fails if the data has not been fully received by the
client. Do you mean that this new issue must be fixed first. Why a new
issue, after all this is the handling of a half-duplex disconnection
on the remote side ?

--

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Follow my comments about half_duplex_close.diff (current latest patch).

+def handle_close(self):
+if not self._closing:
+self._closing = True
+# try to drain the output buffer
+while self.writable() and self.initiate_send() > 0:
+pass
+self.close()

> *Any* while loop should be avoided in the dispatcher. 
> The risk is you keep the poller busy for more than a single loop or,
> at worst, forever.

Right. I will try to change that.


> Also, you expect initiate_send() to return a meaningful value which
> breaks compatibility with existing asynchat code overriding it.

The patch changes also initiate_send() accordingly in asynchat.

--

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> this is the handling of a half-duplex disconnection on the remote
> side ?

Actually this is not the handling of a half-duplex disconnection on the
remote side, but we need a half-duplex disconnection to test it.

--

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12498] asyncore.dispatcher_with_send, disconnection problem + miss-conception

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Attached yet another patch.
This patch does not use a while loop in handle_close() and handles
POLLHUP as suggested by Charles-François. No changes have been made to
both tests (test_half_duplex_close).

--
Added file: http://bugs.python.org/file23609/half_duplex_close_2.diff

___
Python tracker 
<http://bugs.python.org/issue12498>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13372] handle_close called twice in poll2

[Xavier de Gaye]

New submission from Xavier de Gaye :

When use_poll is True, test_handle_close in test_asyncore.py invokes
twice the handle_close() method.

The attached patch:

modifies test_handle_close so that it fails when handle_close() is
called more than once

includes a fix that makes readwrite() behave like poll(): check
that the dispatcher instance is still in the socket map, before
calling the handle_xxx methods

--
components: Library (Lib)
files: handle_close.diff
keywords: patch
messages: 147311
nosy: giampaolo.rodola, josiahcarlson, neologix, stutzbach, xdegaye
priority: normal
severity: normal
status: open
title: handle_close called twice in poll2
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file23634/handle_close.diff

___
Python tracker 
<http://bugs.python.org/issue13372>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12637] logging lastResort handler not ignoring messages less than WARNING

[Xavier de Gaye]

New submission from Xavier de Gaye :

The 'Advanced Logging Tutorial' states about the lastResort handler:
"The handler’s level is set to WARNING, so all events at this and
greater severities will be output."

Python 3.2 does not follow this behavior:

Python 3.2 (r32:88445, Jun 18 2011, 20:30:18)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> root = logging.getLogger()
>>> root.setLevel('DEBUG')
>>> root.warning('warning msg')
warning msg
>>> root.debug('debug msg')
debug msg
>>> 

This is fixed with the attached patch:

Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> root = logging.getLogger()
>>> root.setLevel('DEBUG')
>>> root.warning('warning msg')
warning msg
>>> root.debug('debug msg')
>>>

--
components: Library (Lib)
files: logging_lastResort.patch
keywords: patch
messages: 141085
nosy: xdegaye
priority: normal
severity: normal
status: open
title: logging lastResort handler not ignoring messages less than WARNING
type: behavior
versions: Python 3.2
Added file: http://bugs.python.org/file22748/logging_lastResort.patch

___
Python tracker 
<http://bugs.python.org/issue12637>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10561] The pdb command 'clear bpnumber' may delete more than one breakpoint

[Xavier de Gaye]

New submission from Xavier de Gaye :

Description:

1. When deleting a single breakpoint, all the breakpoints located on
   the line of this breakpoint are also deleted. See the test case
   below.

2. The pdb 'clear' command documentation does not mention that all the
   breakpoints on a line can be deleted with the command:

clear filename:lineno

See the proposed bdb patch and documentation patch below.


Test case:
--
#   File foobar.py   #
def main():
pass

if __name__ == '__main__':
main()
#   Test case   #
xavier$ /usr/local/bin/python2.7
Python 2.7 (r27:82500, Jul 13 2010, 21:30:27)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb, foobar
>>> pdb.run('foobar.main()')
> (1)()
(Pdb) break foobar.main
Breakpoint 1 at /home/xavier/tmp/foobar.py:1
(Pdb) break foobar.main
Breakpoint 2 at /home/xavier/tmp/foobar.py:1
(Pdb) break
Num Type Disp Enb   Where
1   breakpoint   keep yes   at /home/xavier/tmp/foobar.py:1
2   breakpoint   keep yes   at /home/xavier/tmp/foobar.py:1
(Pdb) clear 1
Deleted breakpoint 1
(Pdb) break
(Pdb)
#


Patch:
--
Index: Doc/library/pdb.rst
===
--- Doc/library/pdb.rst (revision 86836)
+++ Doc/library/pdb.rst (working copy)
@@ -239,7 +239,8 @@
Temporary breakpoint, which is removed automatically when it is first hit.  
The
arguments are the same as break.
 
-cl(ear) [*bpnumber* [*bpnumber ...*]]
+cl(ear) [*filename:lineno* | *bpnumber* [*bpnumber ...*]]
+   With a *filename:lineno* argument, clear all the breakpoints at this line.
With a space separated list of breakpoint numbers, clear those breakpoints.
Without argument, clear all breaks (but first ask confirmation).
 
Index: Lib/bdb.py
===
--- Lib/bdb.py  (revision 86836)
+++ Lib/bdb.py  (working copy)
@@ -250,6 +250,12 @@
 list.append(lineno)
 bp = Breakpoint(filename, lineno, temporary, cond, funcname)
 
+def prune_breaks(self, filename, lineno):
+if (filename, lineno) not in Breakpoint.bplist:
+self.breaks[filename].remove(lineno)
+if not self.breaks[filename]:
+del self.breaks[filename]
+
 def clear_break(self, filename, lineno):
 filename = self.canonic(filename)
 if not filename in self.breaks:
@@ -261,10 +267,7 @@
 # pair, then remove the breaks entry
 for bp in Breakpoint.bplist[filename, lineno][:]:
 bp.deleteMe()
-if (filename, lineno) not in Breakpoint.bplist:
-self.breaks[filename].remove(lineno)
-if not self.breaks[filename]:
-del self.breaks[filename]
+self.prune_breaks(filename, lineno)
 
 def clear_bpbynumber(self, arg):
 try:
@@ -277,7 +280,8 @@
 return 'Breakpoint number (%d) out of range' % number
 if not bp:
 return 'Breakpoint (%d) already deleted' % number
-self.clear_break(bp.file, bp.line)
+bp.deleteMe()
+self.prune_breaks(bp.file, bp.line)
 
 def clear_all_file_breaks(self, filename):
 filename = self.canonic(filename)
===

--
components: Library (Lib)
messages: 122652
nosy: xdegaye
priority: normal
severity: normal
status: open
title: The pdb command 'clear bpnumber' may delete more than one breakpoint
type: behavior
versions: Python 2.7

___
Python tracker 
<http://bugs.python.org/issue10561>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28562] test_asyncio fails on Android upon calling getaddrinfo()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

According to my last post on this issue 2 years ago, this test "does not fail 
on android-24-x86_64". This means that it does not fail on API level 24. IMO 
the issue may be closed.

--

___
Python tracker 
<https://bugs.python.org/issue28562>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25172] Unix-only crypt should not be present on Windows.

[Xavier de Gaye]

Xavier de Gaye  added the comment:

test_crypt fails on android following last changes made at 
243a73deee4ac61fe06602b7ed56b6df01e19f27.
The android libc does not have a crypt() function and the _crypt module is not 
built.


generic_x86_64:/data/local/tmp/python $ python
Python 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26)
[Clang 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec7462 
on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import crypt
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/crypt.py", line 6, in 
import _crypt
ModuleNotFoundError: No module named '_crypt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1, in 
  File "/data/local/tmp/python/lib/python3.9/crypt.py", line 11, in 
raise ImportError("The required _crypt module was not built as part of 
CPython")
ImportError: The required _crypt module was not built as part of CPython
>>>


generic_x86_64:/data/local/tmp/python $ python -m test -v test_crypt
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://andro
id.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_3523
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_crypt
test_blowfish_rounds (test.test_crypt.CryptTestCase) ... skipped 'Not supported 
on Windows'
test_crypt (test.test_crypt.CryptTestCase) ... skipped 'Not supported on 
Windows'
test_invalid_rounds (test.test_crypt.CryptTestCase) ... skipped 'Not supported 
on Windows'
test_methods (test.test_crypt.CryptTestCase) ... skipped 'Not supported on 
Windows'
test_salt (test.test_crypt.CryptTestCase) ... skipped 'Not supported on Windows'
test_saltedcrypt (test.test_crypt.CryptTestCase) ... skipped 'Not supported on 
Windows'
test_sha2_rounds (test.test_crypt.CryptTestCase) ... skipped 'Not supported on 
Windows'
test_failure_only_for_windows (test.test_crypt.TestWhyCryptDidNotImport) ... 
FAIL
test_import_failure_message (test.test_crypt.TestWhyCryptDidNotImport) ... FAIL

==
FAIL: test_failure_only_for_windows (test.test_crypt.TestWhyCryptDidNotImport)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_crypt.py", line 16, in 
test_failure_only_for_
windows
self.assertEqual(sys.platform, 'win32')
AssertionError: 'linux' != 'win32'
- linux
+ win32

==
FAIL: test_import_failure_message (test.test_crypt.TestWhyCryptDidNotImport)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_crypt.py", line 19, in 
test_import_failure_message
self.assertIn('not supported', IMPORT_ERROR)
AssertionError: 'not supported' not found in 'The required _crypt module was 
not built as part of CPython'

--

Ran 9 tests in 0.008s

FAILED (failures=2, skipped=7)
test test_crypt failed
test_crypt failed

== Tests result: FAILURE ==

1 test failed:
test_crypt

Total duration: 165 ms
Tests result: FAILURE

--
nosy: +xdegaye

___
Python tracker 
<https://bugs.python.org/issue25172>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38840] incorrect __all__ list in multiprocessing.managers module

[Xavier de Gaye]

New submission from Xavier de Gaye :

On android which is a platform that is missing the shared memory 
implementation, test___all__ fails because 'multiprocessing.managers' has no 
attribute 'SharedMemoryManager' which is listed in __all__.


2|generic_x86_64:/data/local/tmp/python $ python
Python 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) 
[Clang 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec7462 
on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing.shared_memory
Traceback (most recent call last):
  File "", line 1, in 
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/shared_memory.py", 
line 22, in 
import _posixshmem
ModuleNotFoundError: No module named '_posixshmem'
>>> 


2|generic_x86_64:/data/local/tmp/python $ python -m test test___all__
0:00:00 Run tests sequentially
0:00:00 [1/1] test___all__
test test___all__ failed -- Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test___all__.py", line 38, in 
check_all
exec("from %s import *" % modname, names)
AttributeError: module 'multiprocessing.managers' has no attribute 
'SharedMemoryManager'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test___all__.py", line 41, in 
check_all
self.fail("__all__ failure in {}: {}: {}".format(
AssertionError: __all__ failure in multiprocessing.managers: AttributeError: 
module 'multiprocessing.managers' has no attribute 'SharedMemoryManager'

test___all__ failed

== Tests result: FAILURE ==

1 test failed:
test___all__

Total duration: 1.8 sec
Tests result: FAILURE

--
components: Library (Lib)
messages: 356922
nosy: xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: incorrect __all__ list in multiprocessing.managers module
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38840>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38841] [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user

[Xavier de Gaye]

New submission from Xavier de Gaye :

This is the same kind of issue as reported in #28684.


python -m test -v test_asyncio -m 
test_create_datagram_endpoint_existing_sock_unix
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://andro
id.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_6046
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
test_create_datagram_endpoint_existing_sock_unix 
(test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... ERROR
/data/local/tmp/python/lib/python3.9/unittest/case.py:687: ResourceWarning: 
unclosed 
  outcome.errors.clear()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
==
ERROR: test_create_datagram_endpoint_existing_sock_unix 
(test.test_asyncio.test_base_events.BaseEven
tLoopWithSelectorTests)
--
Traceback (most recent call last):
  File 
"/data/local/tmp/python/lib/python3.9/test/test_asyncio/test_base_events.py", 
line 1707, in t
est_create_datagram_endpoint_existing_sock_unix
sock.bind(path)
PermissionError: [Errno 13] Permission denied

--

Ran 1 test in 0.014s

FAILED (errors=1)
test test_asyncio failed
test_asyncio failed

== Tests result: FAILURE ==

1 test failed:
test_asyncio

Total duration: 542 ms
Tests result: FAILURE

--
components: asyncio
messages: 356924
nosy: asvetlov, xdegaye, yselivanov
priority: normal
severity: normal
stage: needs patch
status: open
title: [asyncio] bind() on a unix socket raises PermissionError on Android for 
a non-root user
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38841>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38848] test_compileall fails when the platform lacks a functional sem_open()

[Xavier de Gaye]

New submission from Xavier de Gaye :

See also the related issues:
#32126: [asyncio] test failure when the platform lacks a functional  sem_open()
#28668: instanciation of multiprocessing.Queue raises ImportError in 
test_logging

The test failure on android API 24:

generic_x86_64:/data/local/tmp/python $ python -m test -v test_compileall -m 
test_workers
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://andro
id.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_2579
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_compileall
test_workers (test.test_compileall.CommandLineTestsNoSourceEpoch) ... FAIL
test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch) ... FAIL

==
FAIL: test_workers (test.test_compileall.CommandLineTestsNoSourceEpoch)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_py_compile.py", line 20, 
in wrapper
return fxn(*args, **kwargs)
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 
707, in test_workersself.assertRunOK(self.directory, '-j', '0')
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 
397, in assertRunOK
rc, out, err = script_helper.assert_python_ok(
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", 
line 157, in assert_pyt
hon_ok
return _assert_python(True, *args, **env_vars)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", 
line 143, in _assert_py
thon
res.fail(cmd_line)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", 
line 70, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', 
'-S', '-m', 'compile
all', '/data/local/tmp/python/tmp/tmpc1hy_667', '-j', '0']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", 
line 28, in 
from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' 
(/data/local/tmp/python/lib/python
3.9/lib-dynload/_multiprocessing.cpython-39d.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 192, in 
_run_module_as_main
return _run_code(code, main_globals, None,
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 425, in 

exit_status = int(not main())
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 403, in main
if not compile_dir(dest, maxlevels, args.ddir,
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 91, in 
compile_dir
with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", 
line 555, in __init__
self._call_queue = _SafeQueue(
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", 
line 165, in __init__
super().__init__(max_size, ctx=ctx)
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/queues.py", line 
42, in __init__
self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/context.py", line 
67, in Lock
from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", 
line 30, in 
raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, 
therefore, the required sync
hronization primitives needed will not function, see issue 3770.
---

==
FAIL: test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch)
==  
[38/374]
FAIL: test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_py_compile.py", line 

[issue25172] Unix-only crypt should not be present on Windows.

[Xavier de Gaye]

Change by Xavier de Gaye :


--
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue25172>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38849] test_timestamp_naive fails on android

[Xavier de Gaye]

New submission from Xavier de Gaye :

test_timestamp_naive of test_datetime fails on android API 24:

generic_x86_64:/data/local/tmp/python $ python -m test -v test_datetime -m 
test_timestamp_naive
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_2606
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_datetime
test_timestamp_naive (test.datetimetester.TestDateTime_Pure) ... FAIL
test_timestamp_naive (test.datetimetester.TestDateTimeTZ_Pure) ... FAIL
test_timestamp_naive (test.datetimetester.TestSubclassDateTime_Pure) ... FAIL
test_timestamp_naive (test.datetimetester.TestDateTime_Fast) ... FAIL
test_timestamp_naive (test.datetimetester.TestDateTimeTZ_Fast) ... FAIL
test_timestamp_naive (test.datetimetester.TestSubclassDateTime_Fast) ... FAIL

==
FAIL: test_timestamp_naive (test.datetimetester.TestDateTime_Pure)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_n
aive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

==
FAIL: test_timestamp_naive (test.datetimetester.TestDateTimeTZ_Pure)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_n
aive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

==
FAIL: test_timestamp_naive (test.datetimetester.TestSubclassDateTime_Pure)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_n
aive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

==
FAIL: test_timestamp_naive (test.datetimetester.TestDateTime_Fast)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_n
aive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

==
FAIL: test_timestamp_naive (test.datetimetester.TestDateTimeTZ_Fast)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_naive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

==
FAIL: test_timestamp_naive (test.datetimetester.TestSubclassDateTime_Fast)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/support/__init__.py", line 
1833, in inner
return func(*args, **kwds)
  File "/data/local/tmp/python/lib/python3.9/test/datetimetester.py", line 
2367, in test_timestamp_naive
self.assertEqual(t.timestamp(), 18000.0)
AssertionError: 14400.0 != 18000.0

--

Ran 6 tests in 0.026s

FAILED (failures=6)
test test_datetime failed
test_datetime failed

== Tests result: FAILURE ==

1 test failed:
test_datetime

Total duration: 331 ms
Tests result: FAILURE

--
components: Tests
messages: 356975
nosy: xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: test_timestamp_naive fails on android
type: be

[issue38850] test_largefile fails on android

[Xavier de Gaye]

New submission from Xavier de Gaye :

The failure on andoid API 24:

generic_x86_64:/data/local/tmp/python $ python -m test -v test_largefile -m 
test_it
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://andro
id.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_2626
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_largefile
test_it (test.test_largefile.TestCopyfile) ... ERROR
test_it (test.test_largefile.TestSocketSendfile) ... Exception in thread 
Thread-1:
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/threading.py", line 944, in 
_bootstrap_inner
self.run()
  File "/data/local/tmp/python/lib/python3.9/threading.py", line 882, in run
self._target(*self._args, **self._kwargs)
  File "/data/local/tmp/python/lib/python3.9/test/test_largefile.py", line 193, 
in run
f.write(chunk)
OSError: [Errno 28] No space left on device
ERROR

==
ERROR: test_it (test.test_largefile.TestCopyfile)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_largefile.py", line 160, 
in test_it
shutil.copyfile(TESTFN, TESTFN2)
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 270, in copyfile
_fastcopy_sendfile(fsrc, fdst)
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 163, in 
_fastcopy_sendfile
raise err from None
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 149, in 
_fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '@test_2626_tmp' -> 
'@test_2626_tmp2'

==
ERROR: test_it (test.test_largefile.TestSocketSendfile)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_largefile.py", line 207, 
in test_it
shutil.copyfile(TESTFN, TESTFN2)
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 270, in copyfile
_fastcopy_sendfile(fsrc, fdst)
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 163, in 
_fastcopy_sendfile
raise err from None
  File "/data/local/tmp/python/lib/python3.9/shutil.py", line 149, in 
_fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '@test_2626_tmp' -> 
'@test_2626_tmp2'

==
ERROR: test_it (test.test_largefile.TestSocketSendfile)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_largefile.py", line 207, 
in test_it
client.sendfile(f)
  File "/data/local/tmp/python/lib/python3.9/socket.py", line 483, in sendfile
return self._sendfile_use_sendfile(file, offset, count)
  File "/data/local/tmp/python/lib/python3.9/socket.py", line 400, in 
_sendfile_use_sendfile
raise err from None
  File "/data/local/tmp/python/lib/python3.9/socket.py", line 386, in 
_sendfile_use_sendfile
sent = os_sendfile(sockno, fileno, offset, blocksize)
BrokenPipeError: [Errno 32] Broken pipe

--

Ran 2 tests in 1.207s

FAILED (errors=2)
test test_largefile failed
test_largefile failed

== Tests result: FAILURE ==

1 test failed:
test_largefile

Total duration: 1.4 sec
Tests result: FAILURE

--
components: Tests
messages: 356976
nosy: xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: test_largefile fails on android
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38850>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38841] [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user

[Xavier de Gaye]

Xavier de Gaye  added the comment:

No, it is the SELinux configuration on android devices that forbids binds to 
named UNIX sockets. Changing from a named UNIX socket to an unnamed UNIX socket 
would fix the problem. I don't know if all platforms support unnamed UNIX 
sockets.

The fix in issue #28684 (referenced in the OP) provides a new decorator to skip 
the test for platforms such as android.

--

___
Python tracker 
<https://bugs.python.org/issue38841>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38851] UDPLITE tests fail on android

[Xavier de Gaye]

New submission from Xavier de Gaye :

Attached test_socket.txt is the output of running 'python -m test -v 
test_socket' on android API 24. The 108 tests in error are UDPLITE tests 
introduced in issue #37345.

--
components: Tests
files: test_socket.txt
messages: 356985
nosy: asvetlov, gappleto97, xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: UDPLITE tests fail on android
type: behavior
versions: Python 3.9
Added file: https://bugs.python.org/file48722/test_socket.txt

___
Python tracker 
<https://bugs.python.org/issue38851>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38852] test_recursion_limit in test_threading crashes with SIGSEGV on android

[Xavier de Gaye]

New submission from Xavier de Gaye :

Actually it is the script that is spawned by test_recursion_limit that crashes 
with SIGSEGV on android API 24. Lowering the recursion limit in the script from 
1000 to 100 with sys.setrecursionlimit() fixes the problem.

Here is the error:

generic_x86_64:/data/local/tmp/python $ python -m test -v test_threading -m 
test_recursion_limit   
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 
8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_4603
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_threading
test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL

==
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
--
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_threading.py", line 
1086, in test_recursion_limit
self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error: 

--

Ran 1 test in 0.148s

FAILED (failures=1)
test test_threading failed
test_threading failed

== Tests result: FAILURE ==

1 test failed:
test_threading

Total duration: 354 ms
Tests result: FAILURE

--
components: Tests
messages: 356988
nosy: xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: test_recursion_limit in test_threading crashes with SIGSEGV on android
type: crash
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38841] [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user

[Xavier de Gaye]

Change by Xavier de Gaye :


--
keywords: +patch
pull_requests: +16787
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17294

___
Python tracker 
<https://bugs.python.org/issue38841>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38840] incorrect __all__ list in multiprocessing.managers module

[Xavier de Gaye]

Change by Xavier de Gaye :


--
keywords: +patch
pull_requests: +16788
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17296

___
Python tracker 
<https://bugs.python.org/issue38840>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38848] compileall fails when the platform lacks a functional sem_open()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Changing the title to "compileall fails when the platform lacks a functional  
sem_open()" as the problem lies in the compileall module itself.
Nosying Antoine as the author of issue #36786.


compileall fails on android API 24:

generic_x86_64:/data/local/tmp/python $ python -m compileall --workers=2 .
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", 
line 28, in 
from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' 
(/data/local/tmp/python/lib/python3.9/lib-dynload/_multiprocessing.cpython-39d.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 192, in 
_run_module_as_main
return _run_code(code, main_globals, None,
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 425, in 

exit_status = int(not main())
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 403, in main
if not compile_dir(dest, maxlevels, args.ddir,
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 91, in 
compile_dir
with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", 
line 555, in __init__
self._call_queue = _SafeQueue(
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", 
line 165, in __init__
super().__init__(max_size, ctx=ctx)
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/queues.py", line 
42, in __init__
self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/context.py", line 
67, in Lock
from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", 
line 30, in 
raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, 
therefore, the required synchronization primitives needed will not function, 
see issue 3770.

--
components: +Library (Lib) -Tests
nosy: +pitrou
title: test_compileall fails when the platform lacks a functional  sem_open() 
-> compileall fails when the platform lacks a functional  sem_open()

___
Python tracker 
<https://bugs.python.org/issue38848>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38841] [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user

[Xavier de Gaye]

Change by Xavier de Gaye :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8

___
Python tracker 
<https://bugs.python.org/issue38841>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38848] compileall fails when the platform lacks a functional sem_open()

[Xavier de Gaye]

Change by Xavier de Gaye :


--
keywords: +patch
pull_requests: +16792
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17300

___
Python tracker 
<https://bugs.python.org/issue38848>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38852] test_recursion_limit in test_threading crashes with SIGSEGV on android

[Xavier de Gaye]

Xavier de Gaye  added the comment:

The crash occurs only on debug builds.

See the FreeBSD related issue #37906.

--

___
Python tracker 
<https://bugs.python.org/issue38852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37906] FreeBSD: test_threading: test_recursion_limit() crash with SIGSEGV and create a coredump

[Xavier de Gaye]

Xavier de Gaye  added the comment:

See the android related issue #38852.

--
nosy: +xdegaye

___
Python tracker 
<https://bugs.python.org/issue37906>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38852] test_recursion_limit in test_threading crashes with SIGSEGV on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
keywords: +patch
pull_requests: +16821
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17337

___
Python tracker 
<https://bugs.python.org/issue38852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38852] test_recursion_limit in test_threading crashes with SIGSEGV on android

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Using test and try with _thread.stack_size(new_size), the pthread stack sizes 
must be set to the following minimums to prevent stack overflow and get a 
RecursionError:
  * debug builds:7 Mb
  * no-debug builds: 1 Mb

The default stack size for the main thread does not need to be changed as test 
test_exceptions.ExceptionTests.testInfiniteRecursion is ok for both build types.

PR 17337 sets the thread stack size to 8 Mb for debug builds on android 
platforms.

--

___
Python tracker 
<https://bugs.python.org/issue38852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Change by Xavier de Gaye :


--
pull_requests: +16901
pull_request: https://github.com/python/cpython/pull/17420

___
Python tracker 
<https://bugs.python.org/issue28833>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Xavier de Gaye  added the comment:

PR 17420 fixes cross-compilation of third-party extension modules.

The PYTHON_PROJECT_BASE environment variable is the path to the directory where 
Python has been cross-compiled. It is used by the native python interpreter to 
find the target sysconfigdata module.

For example the following command builds a wheel file to be transfered and 
installed with pip on the target platform, provided the native python 
interpreter and the cross-compiled one both have the wheel package installed:

  $ PYTHON_PROJECT_BASE=/path/to/builddir python setup.py bdist_wheel

--

___
Python tracker 
<https://bugs.python.org/issue28833>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38949] incorrect prefix, exec_prefix in distutils.command.install

[Xavier de Gaye]

New submission from Xavier de Gaye :

In function finalize_options() of Lib/distutils/command/install.py at

https://github.com/python/cpython/blob/575d0b46d122292ca6e0576a91265d7abf7cbc3d/Lib/distutils/command/install.py#L284

(prefix, exec_prefix) is set using get_config_vars(). This may be incorrect 
when Python has been manually copied in another location from the location 
where it has been installed with 'make install'. We should use sys.prefix and 
sy.exec_prefix instead, those values are calculated by getpath.c instead of 
being retrieved from the sysconfigdata module.

--
components: Distutils
messages: 357678
nosy: dstufft, eric.araujo, xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: incorrect prefix, exec_prefix in distutils.command.install
type: behavior
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38949>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38992] testFsum failure caused by constant folding of a float expression

[Xavier de Gaye]

New submission from Xavier de Gaye :

Title: testFsum failure caused by constant folding of a float expression

Description:

Python (Python 3.9.0a1+ heads/master-dirty:ea9835c5d1) is built on a Linux 
x86_64. This native interpreter is used to cross-compile Python (using the same 
source) to Android API 24. Next the installation is done locally to DESTDIR by 
running 'make install' with the env var DESTDIR set and the standard library 
modules are compiled by the native interpreter during this process.  The 
content of DESTDIR is then copied to an arm64 android device (Huawei FWIW). The 
test_math.MathTests.testFsum test fails on the android device with:

AssertionError: -4.309103330548428e+214 != -1.0

This occurs when testing '([1.7**(i+1)-1.7**i for i in range(1000)] + 
[-1.7**1000], -1.0)' in test_values.

Next the test_math.py file is touched on the android device to force 
recompilation of the module and testFsum becomes surprisingly successful.

Investigation:
--
The hexadecimal representation of 1.7**n on x86_64 and arm64 are:
* different for n in (10, 100, 1000)
* equal for n in [0, 9] or 11

on x86_64:
>>> 1.7**10
201.59939004489993
>>> (1.7**10).hex()
'0x1.9332e34080c95p+7'

on arm64:
>>> 1.7**10
201.59939004489996
>>> (1.7**10).hex()
'0x1.9332e34080c96p+7'

The output of the following foo.py module that has been run on x86_64 and arm64 
are attached to this issue:

###
import math, dis

def test_fsum():
x = [1.7**(i+1)-1.7**i for i in range(10)] + [-1.7**10]
return x

y = test_fsum()
print(y)
print(math.fsum(y))
dis.dis(test_fsum)
###

The only difference between both dissasembly of test_fsum() is at bytecode 16 
that loads the folded constant 1.7**10.

Conclusion:
---
The compilation of the expression '[1.7**(i+1)-1.7**i for i in range(1000)] + 
[-1.7**1000]' on x86_64 folds '1.7**1000' to 2.8113918290273277e+230 When the 
list comprehension (the first term of the expression) is executed on arm64, 
then 1.7**1000 is evaluated as 2.8113918290273273e+230.  On arm64 1.7**1000 - 
2.8113918290273277e+230 = -4.309103330548428e+214, hence the AssertionError 
above.

This is confirmed by changing testFsum to prevent constant folding by replacing 
1000 in the testFsum expression with a variable whose value is 1000.  In that 
case the test_math module compiled on x86_64 is successful on arm64. This could 
be a fix for this issue unless this fix would be hiding another problem such as 
.pyc files portability across different platforms and my knowledge of IEEE 754 
is too superficial to answer that point.

--
components: Tests
files: foo.x86_64
messages: 357969
nosy: tim.peters, vstinner, xdegaye
priority: normal
severity: normal
stage: needs patch
status: open
title: testFsum failure caused by constant folding of a float expression
type: behavior
versions: Python 3.9
Added file: https://bugs.python.org/file48762/foo.x86_64

___
Python tracker 
<https://bugs.python.org/issue38992>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38992] testFsum failure caused by constant folding of a float expression

[Xavier de Gaye]

Change by Xavier de Gaye :


Added file: https://bugs.python.org/file48763/foo.arm64

___
Python tracker 
<https://bugs.python.org/issue38992>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38992] testFsum failure caused by constant folding of a float expression

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Yes PR GH-17513 does fix the problem.
Thanks Mark.

--

___
Python tracker 
<https://bugs.python.org/issue38992>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38949] incorrect prefix, exec_prefix in distutils.command.install

[Xavier de Gaye]

Xavier de Gaye  added the comment:

get_config_vars() defined in distutils.sysconfig sets 'prefix' and 
'exec_prefix' using sys.prefix (resp. sys.exec_prefix) on non nt platforms so 
there is no problem. Closing the issue as not a bug.

--
resolution:  -> not a bug
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue38949>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38852] test_recursion_limit in test_threading crashes with SIGSEGV on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue38852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38840] incorrect __all__ list in multiprocessing.managers module

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--

___
Python tracker 
<https://bugs.python.org/issue38840>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38848] compileall fails when the platform lacks a functional sem_open()

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--

___
Python tracker 
<https://bugs.python.org/issue38848>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38851] UDPLITE tests fail on android

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--

___
Python tracker 
<https://bugs.python.org/issue38851>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38850] test_largefile fails on android

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--

___
Python tracker 
<https://bugs.python.org/issue38850>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38849] test_timestamp_naive fails on android

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue38849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25172] Unix-only crypt should not be present on Windows.

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Not interested anymore in android stuff.

--

___
Python tracker 
<https://bugs.python.org/issue25172>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36414] Multiple test failures in GCC and Clang optional builds on Travis CI

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36414>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36758] configured libdir not correctly passed to Python executable

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36758>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35813] shared memory construct to avoid need for serialization between processes

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue35813>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38840] incorrect __all__ list in multiprocessing.managers module

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue38840>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38848] compileall fails when the platform lacks a functional sem_open()

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue38848>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38851] UDPLITE tests fail on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue38851>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38850] test_largefile fails on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue38850>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38849] test_timestamp_naive fails on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
stage: resolved -> 

___
Python tracker 
<https://bugs.python.org/issue38849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25172] Unix-only crypt should not be present on Windows.

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue25172>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28833] cross compilation of third-party extension modules

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue28833>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28190] Cross-build _curses failed if host ncurses headers and target ncurses headers have different layouts

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue28190>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36361] generate correct pyconfig.h when cross-compiling

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36361>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36351] the ipv6type variable in configure.ac may be set incorrectly when cross-compiling

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36351>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36214] AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et al

[Xavier de Gaye]

Change by Xavier de Gaye :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue36214>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36125] Cannot cross-compile to more featureful but same tune

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36125>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27640] add the '--disable-test-suite' option to configure

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue27640>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36141] configure: error: could not find pthreads on your system during cross compilation

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36141>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36145] android arm cross compilation fails, config issue

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue36145>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35997] ImportError: dlopen failed: cannot locate symbol "PyBool_Type"

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue35997>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31046] ensurepip does not honour the value of $(prefix)

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue31046>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26852] add the '--enable-sourceless-distribution' option to configure

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue26852>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35953] crosscompilation fails with clang on android

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue35953>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13886] readline-related test_builtin failure

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue13886>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26855] android: add platform.android_ver()

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue26855>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32203] [ctypes] test_struct_by_value fails on android-24-arm64

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue32203>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32202] [ctypes] all long double tests fail on android-24-x86_64

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue32202>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32461] the first build after a change to Makefile.pre.in uses the old Makefile

[Xavier de Gaye]

Change by Xavier de Gaye :


--
resolution:  -> not a bug
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue32461>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29267] Cannot override some flags in CFLAGS from the command-line

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue29267>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28542] document cross compilation

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue28542>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20211] setup.py: do not add system header locations when cross compiling

[Xavier de Gaye]

Change by Xavier de Gaye :


--
nosy:  -xdegaye

___
Python tracker 
<https://bugs.python.org/issue20211>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com