Charles-François Natali added the comment:
How do we spot any ASAN issues, though? Does ASAN change the process' return
code on errors?
It aborts:
$ cat /tmp/test.c
int main(int argc, char *argv[])
{
int bar[16] = {0};
/* oops */
return bar[16];
}
$ gcc -Wall -fsanitize=address
Charles-François Natali added the comment:
LGTM!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21233
___
___
Python-bugs-list mailing list
Charles-François Natali added the comment:
Depleting /dev/urandom isn't actually a thing. /dev/urandom on all modern
*nix OSs uses a fast PRNG which is secure as long as it has received enough
bytes of initial entropy.
I didn't say deplete /dev/urandom, I said that when reading from
/dev
Charles-François Natali added the comment:
Using os.urandom is the *right* thing to do for getting random in an
application, but the current implementation effectively punishes people who
use it if their application is highly concurrent.
And I argue that this scenario is almost as likely
Charles-François Natali added the comment:
Also I did set mpd_callocfunc to PyMem_Calloc(). 2% slowdown is far
from being a tragic result, so I guess we can ignore that.
Agreed.
The bytes() speedup is very nice. Allocations that took one second
are practically instant now.
Indeed.
Victor
Charles-François Natali added the comment:
I don't think io.DEFAULT_BUFFER_SIZE makes much sense as a heuristic for the
gzip module (or compressed files in general). Perhaps gzip should get its own
DEFAULT_BUFFER_SIZE?
Do you mean from a namespace point of vue, or from a performance point
Charles-François Natali added the comment:
That could make sense, dunno.
Note that the bz2 module uses a harcoded 8K value.
Note that the buffer size should probably be passed to the open() call.
Also, the allocation is quite peculiar: it uses an exponential buffer
size, starting at a tiny
Charles-François Natali added the comment:
Perhaps so, but I think we should open a separate ticket for that
instead of instituting some feature creep here (no matter how
reasonable the concept or its changes would be).
Agreed.
The patch looks good to me, so feel free to commit!
(FWIW, gzip
Charles-François Natali added the comment:
Yes, I'm proposing to enhance this class.
The problem is AFAICT there's currently no way to get a file
descriptor to the underlying /dev/urandom (and I don't know how it
works on Windows).
Also, this would duplicate the work which has already been
Charles-François Natali added the comment:
I read again some remarks about alignement, it was suggested to provide
allocators providing an address aligned to a requested alignement. This topic
was already discussed in #18835.
The alignement issue is really orthogonal to the calloc one, so
Charles-François Natali added the comment:
This issue was opened to be able to use tracemalloc on numpy. I would
like to make sure that calloc is enough for numpy. I would prefer to
change the malloc API only once.
Then please at least rename the issue. Also, I don't see why
everything
Charles-François Natali added the comment:
Note to numpy devs: it would be great if some of you followed the
python-dev mailing list (I know it can be quite volume intensive, but
maybe simple filters could help keep the noise down): you guys have
definitely both expertise and real-life
Charles-François Natali added the comment:
It looks like calloc-3.patch is wrong: it modify _PyObject_GC_Malloc() to
fill the newly allocated buffer with zeros, but _PyObject_GC_Malloc() is not
only called by PyType_GenericAlloc(): it is also used by _PyObject_GC_New
Charles-François Natali added the comment:
I'm with Antoine, it's likely a glibc bug.
We already had a similar issue with fwrite():
http://bugs.python.org/issue17976
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Charles-François Natali added the comment:
__libc_calloc() starts with a check on integer overflow.
Yes, see my previous message:
AFAICT, the two arguments are purely historical (it was used when
malloc() didn't guarantee suitable alignment, and has the advantage of
performing overflow check
Charles-François Natali added the comment:
Thanks for the patch.
That's IMO a good change, but I would only apply it to default, and not
backport it.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21342
Charles-François Natali added the comment:
Like Antoine, I'm really skeptical about the backport: honestly, this change
doesn't bring much in a normal application. To run into the number of open file
descriptors limit (so the scalability aspect), one would need to have *many*
concurrent
Charles-François Natali added the comment:
William, thanks for the benchmarks.
Unfortunately this type of benchmark depends on the hardware (disk, SSD, emmoey
bandwitdh, etc).
So I'd suggest, instead of using an hardcoded value, to simply reuse
io.DEFAULT_BUFFER_SIZE.
That way, if some day
Charles-François Natali added the comment:
@Charles-François: I think your worries about calloc and overcommit are
unjustified. First, calloc and malloc+memset actually behave the same way
here -- with a large allocation and overcommit enabled, malloc and calloc
will both go ahead
Charles-François Natali added the comment:
And your test.py produces the same result. Are you sure you don't have a
ulimit set on address space?
Yep, I'm sure:
$ ulimit -v
unlimited
It's probably due to the exponential over-allocation used by the array
(to guarantee amortized constant cost
Charles-François Natali added the comment:
Dammit, read:
python -c 'bx * (2**48)'
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21233
Charles-François Natali added the comment:
Alright, it bothered me so I wrote a small C testcase (attached),
which calls malloc in a loop, and can call memset upon the allocated
block right after allocation:
$ gcc -o /tmp/test /tmp/test.c; /tmp/test
malloc() returned NULL after 3050MB
$ gcc
Charles-François Natali added the comment:
So yeah, touching pages can affect whether a later malloc returns ENOMEM.
I'm not sure any of this actually matters in the Python case though :-).
There's still no reason to go touching pages pre-emptively just in case we
might write to them
Charles-François Natali added the comment:
Both OOM here (3.11.0-20-generic, 64-bit, Ubuntu).
Hm...
What's /proc/sys/vm/overcommit_memory ?
If it's set to 0, then the kernel will always overcommit.
If you set it to 2, normally you'd definitely get ENOMEM (which is IMO
much nicer than getting
Charles-François Natali added the comment:
Hm...
What's /proc/sys/vm/overcommit_memory ?
If it's set to 0, then the kernel will always overcommit.
I meant 1 (damn, I need sleep).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Charles-François Natali added the comment:
Hm...
What's /proc/sys/vm/overcommit_memory ?
If it's set to 0, then the kernel will always overcommit.
Ah, indeed.
See above, I mistyped: 0 is the default (which is already quite
optimistic), 1 is always.
If you set it to 2, normally you'd
Changes by Charles-François Natali cf.nat...@gmail.com:
--
stage: patch review - commit review
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21362
Charles-François Natali added the comment:
A useful parameter instead would be to support sending only part of the file,
so adding a count argument.
Have you read my patch? This is already provided by the offset parameter.
Of course I read your patch ;-)
I mean I'd like a parameter
Charles-François Natali added the comment:
Updated patch using an anonymous struct.
LGTM!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21207
Charles-François Natali added the comment:
1) I really don't like the use_fallback argument: as a user, I don't
care if it's using sendfile/splice/whatever WIndows uses.
I view this as a channel transfer (like Java's
http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html
Charles-François Natali added the comment:
I was expecting to see such a report :-)
I'm al for the st_ino+st_dev check, it can't hurt.
But everybody must keep in mind that if another thread messes with the
FD between the check and the read, there's nothing we can do
Charles-François Natali added the comment:
Do you have benchmarks?
(I'm not looking for an improvement, just no regression.)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21233
Charles-François Natali added the comment:
In Python 3, arenas are allocated using mmap(), so wherever the arena ends up
in the address space shouldn't matter, should it?
Indeed, although the effect on cache locality isn't clear.
Also, I don't think this solves the problem of having a single
Charles-François Natali added the comment:
So what is the point of _PyObject_GC_Calloc ?
It calls calloc(size) instead of malloc(size), calloc() which can be faster
than malloc()+memset(), see:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html
It will only make
Charles-François Natali added the comment:
Indeed, I think it would make sense to consider this for 3.4, and even 2.7
if we opt for a simple fix.
As for the best way to fix it in the meantime, I'm fine with a buffered
zero-filling (the mere fact that noone ever complained until now probably
Charles-François Natali added the comment:
If I remember correctly the problem is that some OS like linux (and
probably others) do not really allocate space until something is written.
If that's the case then the process may get killed later on when it writes
something in the array.
Yes, it's
Charles-François Natali added the comment:
Also, the FreeBSD man page for mmap() has the following warning:
That's mostly important for real file-backed mapping.
In our case, we don't want a file-backed mmap: we expect the mapping to fit
entirely in memory, so the writeback/read performance
Charles-François Natali added the comment:
Zero-filling mmap's backing file isn't really optimal: why not use truncate()
instead? This way, it'll avoid completely I/O on filesystems that support
sparse files, and should still work on FS that don't.
--
nosy: +neologix
Charles-François Natali added the comment:
This patch can't be reviewed: please re-generate without --git.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21076
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21040
New submission from Charles-François Natali:
Many buildbots are failing with this error:
==
ERROR: idlelib.idle_test.test_warning (unittest.loader.ModuleImportFailure
New submission from Charles-François Natali:
The title says it all: in Modules/hashtable.c, the macro name is
_PY_HASHTABLE_ENTRY_DATA instead of _Py_HASHTABLE_ENTRY_DATA.
Should this be fixed in 3.4?
--
components: Extension Modules
messages: 214570
nosy: haypo, neologix
priority
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - invalid
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20935
Charles-François Natali added the comment:
It's a duplicate of issue #18596, which has already been fixed.
Jeffrey, when you report an issue, please check with the latest version.
Thanks!
--
nosy: +neologix
resolution: - duplicate
stage: - committed/rejected
status: open - closed
New submission from Charles-François Natali:
Adding a compile option to build with ASAN
(https://code.google.com/p/address-sanitizer) could allow us to catch many
memory-related errors (stack/buffer overflows, etc).
Of course, the second step would be to setup buildbots to use this flag
Charles-François Natali added the comment:
Note that ASAN will interfere with the faulthandler's module (since it sets up
its own signal handlers), so if we were to incorporate it into the test suite,
that's something we should look after
Charles-François Natali added the comment:
You could use tcpdump to see what's going on (does the server reply to SYN?).
Note that it might very well be either a firewall setting, or a DoS protection
(some sort of backoff when there are too many SYN within a short interval).
--
nosy
New submission from Charles-François Natali:
This patch updates the socketserver module to use selectors.
It's simpler, will use poll() when available, and also fixes a bug where the
timeout would not be recomputed upon EINTR.
Note that I removed an EINTR-handling test from test_socketserver
Charles-François Natali added the comment:
By the way, could you test with the patch available in issue #21040 ?
It'll use poll() instead of select(): I don't think it'll fix your problem,
but it'll be a nice test anyway :-)
--
___
Python tracker rep
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file34593/socketserver_use_selectors-1.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21040
Charles-François Natali added the comment:
I don't see anything wrong with the code.
Could you try running the test under valgrind.
You must build Python with --with-valgrind, and then:
valgrind --tool=memcheck --suppressions=Misc/valgrind-python.supp test
Charles-François Natali added the comment:
Are you saying that an additional clause for CAN_RAW being defined should
be added around where it is used? Would that sort things out?
Yes.
I'd rather not just revert my change, as that would mean I couldn't
compile the SSL module.
I don't get
Charles-François Natali added the comment:
That AF_CAN was undefined (even though HAVE_LINUX_CAN_H is). This is on
Ubuntu Jaunty, which I use for my Python core development.
How dear...
The latest change should be OK.
--
___
Python tracker rep
Charles-François Natali added the comment:
Vinay, your change just reverted this: http://bugs.python.org/issue20065
Basically, AF_CAN being defined doesn't imply that CAN_RAW is defined.
So compilation will now fail - again - on those hosts.
--
status: closed - open
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file34535/devpoll3_try_again.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18931
Charles-François Natali added the comment:
Could you regenerate it without --git (it doesn't show under the review
tool)?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18931
Charles-François Natali added the comment:
It might be a different test triggering the buffer overflow, but the underlying
cause is the same as #20937.
--
nosy: +neologix
resolution: - duplicate
stage: - committed/rejected
status: open - closed
superseder: - test_socket: buffer
Charles-François Natali added the comment:
Please revert.
To debug, since I guess it's due to a memory corruption because objects are
deallocated while they're still in use, you could try to use valgrind.
Unfortunately, since it's due to a race condition, the overhead will
probably make
Charles-François Natali added the comment:
Well, then it's definitely an OpenBSD bug, and we can't do anything
more than skip the test.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20564
Charles-François Natali added the comment:
It seems that the EINTR should be caught by the standard library in all
cases:
http://bugs.python.org/issue1628205
Yes, it should.
But it's not the case for the socket.create_connection method (verified
in 3.3 and 2.7 source code).
It's
Charles-François Natali added the comment:
Wow, 10 messages in one night...
Could you try to gather all your finding at once, because reading so many
messages in difficult to follow?
GetQueuedCompletionStatus(1 ms)-None took 0.307 ms (monotonic: 0.000 ms)
So basically, on Windows, select(1ms
Charles-François Natali added the comment:
It's not easy because I collect informations from various buildbots and
different virtual machines. But I planned to try to summarize the overall
work done on time in asyncio, select and selectors.
Thanks for the summary.
The first problem
Charles-François Natali added the comment:
STINNER Victor added the comment:
Just so it's clear, those bugs are theoretical: whether you pass
1e-7/1e-10 or 0 to select/kqueue is exactly the same (entering/leaving the
syscall takes some time)...
After many many tests with asyncio (last
Charles-François Natali added the comment:
So this looks like a bug.
I mean an OpenBSD bug.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20564
Charles-François Natali added the comment:
So this looks like a bug.
When running the test, is faulthandler enabled (--timeout option)?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20564
Charles-François Natali added the comment:
I didn't see test_threading failing anymore recently, so I'm closing the
issue. Thanks.
Hm...
Could someone review my message http://bugs.python.org/msg206028?
Because AFAICT, this will lead to random crashes with daemon threads
Charles-François Natali added the comment:
Random thoughts:
- executor-created threads are daemon (unfortunately)
- see issue #19466: I think that the change to clear the frame of daemon
threads was a mistake
--
___
Python tracker rep
Charles-François Natali added the comment:
Should be fixed now.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20065
Charles-François Natali added the comment:
Just a quick comment on the patch: in as_completed(), if the future is
cancelled or finished, control is yielded back to the caller with the
condition's lock held.
As a general rule, libraries should not yield control to the caller with a lock
held
Charles-François Natali added the comment:
Note that the context manager will be called in this case to release the
lock before f is yielded to the caller.
class MiniContext():
def __init__(self):
pass
def __enter__(self):
print('Hello')
def __exit__(self
Charles-François Natali added the comment:
I'm asking why it's 3 seconds by default in the tests?
Could we modify the value to 6 instead of 3, is it acceptable?
No, see this comment:
self.assertRaises(KeyboardInterrupt, lock.acquire, timeout=5)
dt = time.time() - t1
# Checking
Charles-François Natali added the comment:
How sure are you? Suppose I use poll() with a 0.5 msec timeout. This
presumably gets rounded up to 1 msec. But if the system clock has e.g. a
10
msec resolution, won't this still wait 0 msec? Or will it wait until the
next tick occurs, which could
Charles-François Natali added the comment:
Thanks for the detailed report.
connections.py from multiprocessing has been rewrittend between 3.2
and 3.3 but I can't see what's wrong in the way it has been done,
basic socket options seem to be exactly the same.
Indeed
Charles-François Natali added the comment:
select() and kqueue() are able to sleep less than 1 ms. Using a slack of
1 ms would reduce the accuracy. I don't see why we should limit the
accuracy. Why 1 ms? Because of poll/epoll? What about Windows and its
resolution of 15.6 ms?
Well, under 1 ms
Charles-François Natali added the comment:
Updated patch with the 3.2 heuristic for deciding when to concatenate.
LGTM.
For posterity, a quick benchmark performing 100 trivial remote calls:
Before:
$ ./python ~/test_manager.py
8.202659845352173
After:
$ ./python ~/test_manager.py
Charles-François Natali added the comment:
To solve a performance issue in asyncio, I added a new resolution
attribute to selectors.BaseSelector and a new _granularity attribute to
asyncio.BaseEventLoop. If I understood correctly, Charles-François (author
and so maintainer of the new selectors
Charles-François Natali added the comment:
Shouldn't this be fixed in the C implementation of the select module or
in selectors.py? It seems likely that the exact range might be different
for each syscall and possibly per OS or even OS version.
Agreed: if we want to fix this, it should be done
Charles-François Natali added the comment:
BreamoreBoy: why did you remove Arfrever from this issue?
Noisy lists members are sorted by alphabetical order: since Arfrever comes just
before BreamoreBoy, I assume his fingers tripped ;-)
--
nosy: +Arfrever, neologix
Charles-François Natali added the comment:
Well, now that timeouts are properly rounded, the granularity is useless.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20311
Charles-François Natali added the comment:
I don't think so. Please read again the issue #20452, for example this
message:
http://bugs.python.org/issue20452#msg209772
Ok, it looks better: waiting 99.9 ms took 99.6 ms and 99.9 ms, and waiting
9.9 ms took 9.7 ms. So as I said, the granularity
Charles-François Natali added the comment:
Once again: a slight early wakeup isn't an issue
That's your opinion, but I disagree.
Please open a new issue with a patch, or reopen at least this issue because
it is now closed.
I already spent to much time on this issue. Buildbots are now
Charles-François Natali added the comment:
It looks like select() and poll() in Linux 2.6.28 has a resolution
of 1/HZ, where HZ can be retrieved from os.sysconf('SC_CLK_TCK'). Since
Linux 2.6.28, hrtimers are now used for timeouts.
Attached patch improves the resolution field
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - invalid
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20418
Charles-François Natali added the comment:
The culprint isn't Python, but the libc:
$ ./python -c import socket; print(socket.getaddrinfo('836937931829', 80,
socket.AF_INET, 0, socket.SOL_TCP))
Traceback (most recent call last):
File string, line 1, in module
socket.gaierror: [Errno -2
Charles-François Natali added the comment:
If the patch is accepted, my changes on Python 3.3 should also be
reverted.
I'm sorry, but I'm not convinced.
The selector's granularity is an implementation detail, and I don't think
it should be exposed.
Furthermore, it's not a mere function
Charles-François Natali added the comment:
LGTM.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20331
___
___
Python-bugs-list
Charles-François Natali added the comment:
Once again, what's wrong with your initial approach of ceiling the
timeout?
It looks like changing the rounding method doesn't solve anything.
selector.select(timeout) may still take less than timeout, so it
doesn't give any guarantee.
But what
Charles-François Natali added the comment:
But what problem does it cause if, once in a while, the call takes less
than the passed timeout?
If that's the case, you'll simply perform another loop, an wake up 1ms
later, that's all.
perform another loop is my problem. If possible, I would
Charles-François Natali added the comment:
==
FAIL: test_timeout_rounding (test.test_epoll.TestEPoll)
--
Traceback (most recent call last):
File
/srv
Charles-François Natali added the comment:
Sorry for the delay, I had completely forgotten this issue.
Igor, could you try the patch attached?
--
keywords: +patch
Added file: http://bugs.python.org/file33664/af_can_define.diff
___
Python tracker rep
Charles-François Natali added the comment:
Ah? The manual page of epoll_wait() says:
The timeout argument specifies the minimum number of milliseconds that
epoll_wait() will block. (This interval will be rounded up to the system
clock granularity, and kernel scheduling delays mean
Charles-François Natali added the comment:
Did you read my Tulip? Maybe I didn't explain correctly.
No, it was quite clear, and I think I understand the original issue:
calling epoll_wait(0) when passed a timeout of 0.9ms was bad, because it
led to busy-loop during 0.9ms.
But here, that's
Charles-François Natali added the comment:
AFAICT, this also affects poll().
Although it's supposed to be passed an integer, passing a float will result
in a truncation towards 0:
$ strace -e poll python -c import select; p = select.poll(); p.poll(0.9)
poll(0x23321b0, 0, 0
Charles-François Natali added the comment:
Just so it's clear, those bugs are theoretical: whether you pass 1e-7/1e-10 or
0 to select/kqueue is exactly the same (entering/leaving the syscall takes some
time)...
--
___
Python tracker rep
Changes by Charles-François Natali cf.nat...@gmail.com:
--
assignee: - neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20065
Charles-François Natali added the comment:
Hum...
Correct me if I'm wrong, but destroying the thread state of daemon threads
while they're running is really a bad idea in fact: for example, if
warnings are now emitted for unclosed file objects, this means that the
file object, and all associated
Charles-François Natali added the comment:
Here's a patch.
--
Added file: http://bugs.python.org/file33113/makefile_ast_h.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19965
Charles-François Natali added the comment:
Since the patch doesn't use O_EXCL for the temporary file, it suffers
from the exact same race condition as the current code. It should be
using the x open mode.
--
nosy: +neologix
___
Python tracker rep
Charles-François Natali added the comment:
Since the patch doesn't use O_EXCL for the temporary file, it suffers
from the exact same race condition as the current code.
This does not make race condition. Only one thread writes .tmp files.
The problem is that other threads can read
Charles-François Natali added the comment:
Actually no, I don't understand this patch: if the makefile was correct, C
files depending on Include/Python-ast.h and Python/Python-ast.c shouldn't
be built before those files have been generated by asdl_c.py.
The problem doesn't lie in the non
201 - 300 of 1861 matches
Mail list logo
