New submission from Alexander Mohr:
it would be REALLY nice (and REALLY easy) to add a parameter: exist_ok and pass
this to os.makedirs with the same parameter name so you can use copytree to
append a src dir to an existing dst dir.
--
components: Library (Lib)
messages: 212691
nosy
Alexander Mohr added the comment:
awesome, thanks so much!!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20849
___
___
Python-bugs-list
Alexander Mohr added the comment:
how about instead we rename the new parameter to dirs_exists_ok or something
like that since the method already allows for existing files.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Alexander Mohr added the comment:
I personally dont think this is worth investing the time for a discussion.
If the maintainers dont want to accept this or a minor variation without a
discussion ill just keep my local monkeypatch :) thanks again for the
quick patch Elias!
On Mar 8, 2014 4:03 PM
Alexander Mohr added the comment:
Ya. The original request I think is ok because by allowing that flag it
will replace files and dirs.
On Mar 14, 2014 7:15 PM, R. David Murray rep...@bugs.python.org wrote:
R. David Murray added the comment:
I don't know what the method already allows
Alexander Mohr added the comment:
btw, I believe the solution is as simple as stated as that's what I'm doing
locally and its behaving exactly as intended.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20849
Alexander Mohr added the comment:
Sorry for being obscure before, it was hard to pinpoint. I think I just
figured it out! I had code like this in a subprocess:
def worker():
while True:
obj = self.queue.get()
# do work with obj using asyncio http module
def producer
Alexander Mohr added the comment:
clarification, adding the fut.done() check, or monkey patching:
orig_sock_connect_cb =
asyncio.selector_events.BaseSelectorEventLoop._sock_connect_cb
def _sock_connect_cb(self, fut, sock, address):
if fut.done(): return
return orig_sock_connect_cb(self
Alexander Mohr added the comment:
Actually, I just realized I had fixed it locally by changing the callback to
the following:
429 def _sock_connect_cb(self, fut, sock, address):
430 if fut.cancelled() or fut.done():
431 return
so a fix is still needed, and I also
Alexander Mohr added the comment:
I'm going to close this as I've found a work-around, if I find a better
test-case I'll open a new bug.
--
resolution: -> later
status: open -> closed
___
Python tracker <rep...@bugs.python.o
New submission from Alexander Mohr:
asyncio.selector_events.BaseSelectorEventLoop._sock_connect_cb is a callback
based on the selector for a socket. There are certain situations when the
selector triggers twice calling this callback twice, resulting in an
InvalidStateError when it sets
Alexander Mohr added the comment:
attaching my simplified testcase and logged an aiohttp bug:
https://github.com/KeepSafe/aiohttp/issues/633
--
Added file: http://bugs.python.org/file41018/test_app.py
___
Python tracker <rep...@bugs.python.
Alexander Mohr added the comment:
btw want to thank you guys for actively looking into this, I'm very grateful!
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
self.queue is not an async queue, as I stated above its a multiprocessing
queue. This code is to multiplex a multiprocessing queue to a async queue.
--
___
Python tracker <rep...@bugs.python.org>
Alexander Mohr added the comment:
Perhaps I'm doing something really stupid, but I was able to reproduce the two
issues I'm having with the following sample script. If you leave the monkey
patch disabled, you get the InvalidStateError, if you enable it, you get the
ServerDisconnect errors
Alexander Mohr added the comment:
adding support for internal queue size is critical to avoid chewing through all
your memory when you have a LOT of tasks. I just hit this issue myself. If we
could have a simple parameter to set the max queue size this would help
tremendously
Alexander Mohr added the comment:
update: its unrelated to the number of sessions or SSL, but instead to the
number of concurrent aiohttp requests. When set to 500, I get the error, when
set to 100 I do not.
--
___
Python tracker <
Alexander Mohr added the comment:
I'm not sure if you guys are still listening on this closed bug but I think
I've found another issue ;) I'm using python 3.5.1 + asyncio 3.4.3 with the
latest aiobotocore (which uses aiohttp 0.21.0) and had two sessions (two
TCPConnectors), one doing
Alexander Mohr added the comment:
any chance if this getting into 3.5.2? I have some gross code to get around it
(setting global properties)
--
nosy: +thehesiod
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
any updates on this? I think this would be perfect for
https://github.com/aio-libs/aiobotocore/issues/31
--
nosy: +thehesiod
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
updating to make default the error case (madis)
--
Added file: http://bugs.python.org/file46470/scratch_1.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Changes by Alexander Mohr <thehes...@gmail.com>:
Removed file: http://bugs.python.org/file46469/scratch_1.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
New submission from Alexander Mohr:
with the attached code note how HttpClient.connection_lost callback is never
called for the madis server. The madis server is an apache server, I tried with
the OSX apache server and could not reproduce the issue so it seems something
particular about
Alexander Mohr added the comment:
Thanks so much for the patch!
may want to change spelling of what was supposed to be "shutdown" =) Also
think it's worth a comment stating why it's needed? Like certain Apache
servers were noticed to not complete the SSL shutdo
Changes by Alexander Mohr <thehes...@gmail.com>:
Removed file: http://bugs.python.org/file46322/exit_stack.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alexander Mohr added the comment:
created gist: https://gist.github.com/thehesiod/b8442ed50e27a23524435a22f10c04a0
I've now updated the imple to support both __aenter__/__aexit__ and
__enter__/__exit__ so I don't need two ExitStacks
--
___
Python
Alexander Mohr added the comment:
I believe this is now worse due to https://github.com/python/asyncio/pull/452
before I was able to simply create a new run loop from sub-processes however
you will now get the error "Cannot run the event loop while another loop is
running".
New submission from Alexander Mohr:
ExitStack is a really useful class and would be a great to have an async
version. I've gone ahead and created an implementation based on the existing
Python 3.5.2 implementation. Let me know what you guys think. I think it would
be possible to combine
Alexander Mohr added the comment:
Thanks for the feedback Nick! If I get a chance I'll see about refactoring my
gist into a base class and two sub-classes with the async supporting non-async
but not vice-versa. I think it will be cleaner. Sorry I didn't spend too much
effort on the existing
Alexander Mohr added the comment:
ya I did a monkey patch which resolved it.
if sys.platform == 'darwin':
import botocore.vendored.requests.utils, urllib.request
botocore.vendored.requests.utils.proxy_bypass =
urllib.request.proxy_bypass_environment
Alexander Mohr added the comment:
I'm sure it would work, I just wanted a solution that didn't changes to our
build infrastructure. btw if we're marking this as a duplicate of the other
bug, can we update the other bug to say it affects python3.x as well? Thanks!
--
status: pending
Alexander Mohr added the comment:
interestingly I haven't been able to get this to crash in a separate test app.
There must be either timing related to some interaction with another module.
let me know how you guys would like to proceed. I can definitely reproduce it
consistently in our
New submission from Alexander Mohr:
I have a unittest which spawns several processes repeatedly. One of these
subprocesses uses botocore, which itself uses the above two methods through the
calls proxy_bypass and getproxies. It seems after re-spawning the methods a
few times the titled
Alexander Mohr added the comment:
ok I've updated the gist with a base class and sync + async sub-classes. The
way it worked out I think is nice because we can have the same method names
across both sync+async. Let me know what you guys think! btw, it seems
Alexander Mohr added the comment:
adding valgrind log of 3.5.3 on debian: jessie
--
Added file: http://bugs.python.org/file46750/valgrind.log.gz
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
interestingly the valgrind run doesn't show a leak in the profile
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
New submission from Alexander Mohr:
When upgrading to 3.5.3 we noticed that the requests module was leaking memory
rather quickly. This led to me logging the issue:
https://github.com/kennethreitz/requests/issues/3933. After more investigation
I've found that the leak is caused by the raw
Alexander Mohr added the comment:
validated 3.6 in fedora is affected as well, see github bug for charts. So it
seems all 3.5.3+ versions are affected. I'm guessing it was introduced in one
of the SSL changes in 3.5.3:
https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-3
Alexander Mohr added the comment:
yes, in the gist I created you can switch between the various clients, by
default right now it uses raw sockets.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
@pitrou: sys.getallocatedblocks does not seem to increase
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
ya, my sample script hits google.com <http://google.com/>, it's pretty fast.
It just does a "HEAD".
> On Apr 11, 2017, at 9:14 AM, Antoine Pitrou <rep...@bugs.python.org> wrote:
>
>
> Antoine Pitrou added the comment:
Alexander Mohr added the comment:
the interesting part is it doesn't leak with a local https server, it appears
to need to be a remove server.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
see graphs here: https://github.com/kennethreitz/requests/issues/3933, x-axis
is number of requests not what it says (seconds).
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
awesome! Thanks for finding a proposing fix pitrou! btw I found an example of
freeing this structure here:
http://www.zedwood.com/article/c-openssl-parse-x509-certificate-pem
--
___
Python tracker <
Alexander Mohr added the comment:
another core had a different gc object:
$1 = {ob_base = {ob_base = {_ob_next = 0x7f801eac3158, _ob_prev =
0x7f801eab95a0, ob_refcnt = 41, ob_type = 0x7f80238e76e0 },
ob_size = 0}, tp_name = 0x7f801e8967af "_asyncio.Task", tp_basicsize = 128,
tp_it
Alexander Mohr added the comment:
I'm hoping this is the fix:
--- Modules/_asynciomodule.c.orig 2017-07-31 12:16:16.0 -0700
+++ Modules/_asynciomodule.c2017-07-31 13:08:52.0 -0700
@@ -953,15 +953,18 @@
FutureObj_dealloc(PyObject *self)
{
FutureObj *fut
Alexander Mohr added the comment:
oh, so this is looking like an asyncio issue, the "gc" that is causing the
crash is:
(gdb) print *FROM_GC(gc)->ob_type
$8 = {ob_base = {ob_base = {_ob_next = 0x7f80c8aafc88, _ob_prev =
0x7f80c8aafd00, ob_refcnt = 7, ob_type = 0x7f80cd8c86e0 },
Alexander Mohr added the comment:
so I just discovered that the object that has the zero refcount has the same
tp_dealloc:
(gdb) print *FROM_GC(gc)->ob_type
$8 = {ob_base = {ob_base = {_ob_next = 0x7f80c8aafc88, _ob_prev =
0x7f80c8aafd00, ob_refcnt = 7, ob_type = 0x7f80cd8c86e0 }, ob_s
Alexander Mohr added the comment:
omg I just realized I need the default dict one too, great investigation work!
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
New submission from Alexander Mohr:
I have a project in a prod environment which heavily uses asyncio and a
threadpool. It uses the threadpool to run CPU heavy tasks (in this case
populating a defaultdict) to avoid blocking the main thread (no async code in
thread). For some time now my
Alexander Mohr added the comment:
btw I've seen this issue in 3.5.2 + 3.6.2 on debian jessie + stretch
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
bad news, I just got a crash in the same place (updating defaultdict) after
running for a week with the fixes from this and inada naoki's patches. I think
the threadpool may be leaking threads too as I had > 40 threads after running
for a week when I
Alexander Mohr added the comment:
I'm tracking something very similar issue to this in bug:
http://bugs.python.org/issue31061 Given its similarities, anyone have any
ideas? Based on the second callstack I'm starting to think this is an issue
with defaultdict
--
nosy: +thehesiod
Alexander Mohr added the comment:
btw got slightly difference stacktrace on second core file
--
Added file: http://bugs.python.org/file47051/python crash2.txt
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
the problem with this crash is that it only happens periodically in our prod
environment :( If I try running the exact same docker container with the same
inputs locally it doesn't reproduce, so frustrating. I've created a whole
workflow now for deploying
Alexander Mohr added the comment:
so looks like disabling the _asyncio module just caused the crash to happen
less often, closing and will continue investigating after a get a core file
--
stage: -> resolved
status: open -> closed
___
Alexander Mohr added the comment:
ok got a full debug core file, let me know what other information I can provide.
--
status: closed -> open
Added file: http://bugs.python.org/file47049/python crash.txt
___
Python tracker <rep...@bugs.pyth
Alexander Mohr added the comment:
this is the comment on the assert:
/* Python's cyclic gc should never see an incoming refcount
* of 0: if something decref'ed to 0, it should have been
* deallocated immediately at that time.
* Possible cause (if the assert triggers): a tp_dealloc
Alexander Mohr added the comment:
hmm, how would I do that? btw I'm not 100% sure this is due to asyncio.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
ok, created: https://github.com/python/cpython/pull/2966
there are some other deallocs in there, mind verifying the rest?
--
pull_requests: +3014
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Alexander Mohr added the comment:
I suggest any places that don't need the calls should have comments so that
future reviewers know why.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
actually another idea: could the PR for this also update
https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_dealloc to
mention about these macros and when they should be used? That, along with all
the other locations correctly calling
Alexander Mohr added the comment:
should the base method which calls tp_dealloc do this? Maybe can kill all
birds with one stone.
--
nosy: +thehesiod
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
I've verified that this along with the changes in 31095 resolve the crashes
I've been seeing in our production environment
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
hmm, may be my fault due to docker image tagging issue. Will redeploy and
update if the issue persists. If I don't reply again sorry for the noise.
--
___
Python tracker <rep...@bugs.python.org>
Changes by Alexander Mohr <thehes...@gmail.com>:
--
nosy: +thehesiod
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30698>
___
_
Alexander Mohr added the comment:
looks like these were meant to be internal methods, retracting new issues
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/iss
Changes by Alexander Mohr <thehes...@gmail.com>:
--
nosy: +thehesiod
versions: +Python 3.6, Python 3.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.o
Alexander Mohr added the comment:
from 3.6:
>>> AddrlistClass('John Smith <john.smith(comment)@example.org>').getcomment()
''
>>> AddrlistClass('John Smith <john.smith(comment)@example.org>').getdomain()
Alexander Mohr added the comment:
my vote is yes due to the defaultdict issue. We were hitting this in our prod
env
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alexander Mohr added the comment:
let me know if I need to do anything
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29302>
___
__
Change by Alexander Mohr <thehes...@gmail.com>:
--
type: -> behavior
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33565>
___
__
New submission from Alexander Mohr <thehes...@gmail.com>:
while investigating https://github.com/boto/botocore/issues/1464 I used
tracemalloc (like I've done before in 3.5.2) to try to figure out where the
leak was. To my surprise tracemalloc listed stacks that didn't make any
Alexander Mohr <thehes...@gmail.com> added the comment:
I realize it doesn't track leaks, it's a tool to help find leaks when used
correctly :) This example should be similar to using the compare snapshots
mechanism as I start tracking from a stable point (several iterations in after
Alexander Mohr <thehes...@gmail.com> added the comment:
here's another problem, if I change that function signature to:
def test(s3_client):
try:
method = getattr(s3_client, sys.argv[1])
method(Bucket='archpi.dabase.com', Key='doesnotexist')
Alexander Mohr <thehes...@gmail.com> added the comment:
so based on my last comment I just realized we could easily see if something
was amiss by comparing results from python 3.5.2 to 3.6.5 and low and behold
the callstack in question does not appear in the tracemalloc results from
Alexander Mohr <thehes...@gmail.com> added the comment:
actually it does show in 3.5.2, it doesn't show when using get_object, but it
does when using head_object, and they both throw the same exception, so weird.
--
___
Python tracke
New submission from Alexander Mohr <thehes...@gmail.com>:
I'm seeing a lot of leaks via valgrind against the hashlib module. It appears
that it's calling OpenSSL_add_all_digests(); on init, and never calling the
corresponding EVP_Cleanup on free:
https://www.openssl.org/docs/man1.1.0/
Alexander Mohr <thehes...@gmail.com> added the comment:
that's not going to affect
http://pytracemalloc.readthedocs.io/api.html#get_traced_memory. There is no
filter for that :)
as to your sum that's exactly what my original callstack lists:
21 memory blocks: 4.7 KiB
this means 21
Alexander Mohr <thehes...@gmail.com> added the comment:
I believe your method is flawed, when enabling tracemalloc it's going to be
using memory as well to store the traces. I still believe you need to use the
method I mentioned and further even if we don't take into account the total
Alexander Mohr <thehes...@gmail.com> added the comment:
INADA Naoki: Unfortunately you'll need to use credentials from a free AWS
account: https://aws.amazon.com/free/. Then create a credentials file in
~/.aws/credentials:
https://docs.aws.amazon.com/cli/latest/userguide/cli-
Alexander Mohr <thehes...@gmail.com> added the comment:
ok fair enough, however a caveat is that it's not necessarily end of function
as I was able to expand the function by a few more lines and the callstack
stayed the same, however I'm guessing the interpreter was smart enough to
r
Alexander Mohr <thehes...@gmail.com> added the comment:
this is how my friends talk here, see:
https://english.stackexchange.com/questions/11816/is-guy-gender-neutral
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Alexander Mohr <thehes...@gmail.com> added the comment:
of course, sorry, I meant in a gender neutral way
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Alexander Mohr <thehes...@gmail.com> added the comment:
yes, memory does go up. If you click the botocore bug link you'll see a graph
of memory usage over time.
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Alexander Mohr <thehes...@gmail.com> added the comment:
here's a version that tries to do something similar but does not reproduce the
issue
--
Added file: https://bugs.python.org/file47602/tracemalloc_test2.py
___
Python tracke
Alexander Mohr <thehes...@gmail.com> added the comment:
closing as I'm not quite sure this is right
--
stage: -> resolved
status: open -> closed
___
Python tracker <rep...@bugs.python.org>
<https://bugs.
Alexander Mohr <thehes...@gmail.com> added the comment:
not fixing this means that 3.6 slowly leaks for many people in prod. It's not
often possible to fix all the warnings on large dynamic applications, I highly
suggest finding a way to get this into 3.6. I bet there are
Alexander Mohr <thehes...@gmail.com> added the comment:
I'll try with that fix and see if the leak is fixed, is the reasoning that if
the warning happens after the try/except scope and that's why the callstack has
it?
--
___
Python tracke
Alexander Mohr <thehes...@gmail.com> added the comment:
Ok I've verified that the patch does indeed fix the leak detected. Thank you
very much INADA for knowing that there was a leak in the warnings module, I
would have never guessed, especially given the tracemalloc stack given
Alexander Mohr <thehes...@gmail.com> added the comment:
actually, thinking about this more on my way to work, this should NOT be
closed. The callstack I initially mentioned still has no explanation and we
now know is not correct. It should either have listed something related to
wa
Alexander Mohr <thehes...@gmail.com> added the comment:
oh for second example I meant something like this:
>>> class Foo: pass
>>> Foo.organizer = None
>>> Foo.blah = Foo
>>> Foo.blah.organizer = None
>>> Foo.blah.org
New submission from Alexander Mohr <thehes...@gmail.com>:
Currently subscriting a attribute that's None reports the following:
>>> class Foo: pass
>>> Foo.organizer = None
>>> Foo.organizer['start']
Traceback (most recent call last):
File ""
Alexander Mohr <thehes...@gmail.com> added the comment:
@grzgrzgrz3, does this resolve the issue in https://bugs.python.org/issue29406
? I'm guessing you based this PR on that issue. If so I'd like it merged ASAP
as otherwise our prod services will be incompatible with all future
Alexander Mohr <thehes...@gmail.com> added the comment:
my understanding is that the PR in https://bugs.python.org/issue30698 fixes
this issue no? If so can we get it merged?
--
___
Python tracker <rep...@bugs.python.org>
<https://
New submission from Alexander Mohr <thehes...@gmail.com>:
currently gather/wait allow you to return on the first exception and leave the
children executing.
A very common use case that I have is of launching multiple tasks, and if any
of them fail, then all should fail..otherwise the
Change by Alexander Mohr <thehes...@gmail.com>:
--
nosy: +thehesiod
versions: +Python 3.5
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python
Alexander Mohr <thehes...@gmail.com> added the comment:
sorry if my report is confusing, the issue is when run in a debian:stretch
docker container on an OSX host, so running this: docker run --rm -ti
docker:stretch on osx. So if you have access to an OSX machine and have docker
r
Alexander Mohr <thehes...@gmail.com> added the comment:
here's how to repro, download fresh debian:stretch image, then install reqs for
python:
apt-get update && apt-get install curl build-essential libssl-dev libffi-dev
libmemcached-dev zlib1g-dev
install pyenv-installer:
Alexander Mohr <thehes...@gmail.com> added the comment:
btw there are some other tests that fail too after removing that test like:
test test_tokenize failed -- Traceback (most recent call last):
File "/build/Python-3.6.3/Lib/test/test_tokenize.py", line 1557, in
t
1 - 100 of 121 matches
Mail list logo
