Marco Paolini added the comment:
I finally wrapped my head around this. I wrote a (simpler) script to get a
better picture.
What happens
-
When a consumer task is first istantiated, the loop holds a strong reference to
it (_ready)
Later on, as the loop starts, the consumer task
Marco Paolini added the comment:
Asking the user to manage strong refs is just passing the potential
leak issue outside of the standard library. It doesn't really solve anything.
If the user gets the strong refs wrong he can either lose tasks or
leak memory.
If the standard library gets
Marco Paolini added the comment:
So you are changing your mind and withdrawing your option #1.
I think option #1 (tell users to keep strong refs to tasks) is
OK but option #2 is better.
Yes, I changed my mind ;)
--
___
Python tracker rep
Marco Paolini added the comment:
Submitted a first stab at #2. Let me know what you think.
If this works we'll have to remove the test_gc_pending test and then maybe even
the code that now logs errors when a pending task is gc'ed
--
Added file: http://bugs.python.org/file36408
Marco Paolini added the comment:
I don't understand how keeping a strong refrence would fix anything. You
only provided one example (async-gc-bug.py) which uses Queue objects but
keep weak references to them. Keeping strong references to tasks is not the
right fix. You must keep strong
Marco Paolini added the comment:
Sorry for keeping this alive.
Take a look at the `wait_for.py` just submitted in the unrelated #22448: no
strong refs to the tasks are kept. Tasks remain alive only because they are
timers and the event loop keeps strong ref.
Do you think my proposed patch
Marco Paolini added the comment:
I think there is an issue in the way you designed your cleanup logic. So I
think this issue is invalid.
Usually, the code (funcion, class, ...) that *opens* the file should also be
resposible of closing it.
option 1) the caller opens and closes the file
Marco Paolini added the comment:
KeyboardInterrupt is not handled gently by asyncio (see
https://groups.google.com/d/msg/python-tulip/sovg7EIBoXs/m7U-0UXqzSQJ)
you could cancel all tasks in the signal handler:
...
def sig_interrupt():
print('interrupt')
for task
Marco Paolini added the comment:
Linking related issues http://bugs.python.org/issue24721
http://bugs.python.org/issue24667 and http://bugs.python.org/issue24685
--
nosy: +mpaolini
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Changes by Marco Paolini markopaol...@gmail.com:
--
nosy: +mpaolini
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23670
___
___
Python-bugs-list
Changes by Marco Paolini markopaol...@gmail.com:
--
nosy: +mpaolini
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23496
___
___
Python-bugs-list
Marco Paolini added the comment:
Attaching a patch. Does it make any sense?
--
keywords: +patch
nosy: +mpaolini
Added file: http://bugs.python.org/file40353/issue24891.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Changes by Marco Paolini <markopaol...@gmail.com>:
Removed file: http://bugs.python.org/file40353/issue24891.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Marco Paolini added the comment:
ops wrong patch... trying again.
--
Added file: http://bugs.python.org/file40354/issue24891.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Marco Paolini added the comment:
@haypo, yeah, definitely better than mine! All good for me.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Marco Paolini added the comment:
@haypo thanks for the quick review. This new issue24891_2.patch covers all of
the points you raised except the "check exception type" which I am still
figuring out.
--
Added file: http://bugs.python.org/file40355/issue248
Marco Paolini added the comment:
THe guys developing uvloop say their implementation is already quite fast [1].
Is it worth integrating it?
[1] https://github.com/MagicStack/uvloop
--
nosy: +mpaolini
___
Python tracker <rep...@bugs.python.
New submission from Marco Paolini :
I analysed the performance of json.loads in one production workload we have.
Spy-py tells me the majority of time is spent into C json module (see
events.svg)
Digging deeper, linux perf tells me hottest loop (where 20%+ of the time is
spent
Change by Marco Paolini :
--
keywords: +patch
pull_requests: +14547
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/14752
___
Python tracker
<https://bugs.python.org/issu
Change by Marco Paolini :
--
nosy: +ezio.melotti, rhettinger
___
Python tracker
<https://bugs.python.org/issue37587>
___
___
Python-bugs-list mailing list
Unsub
Marco Paolini added the comment:
Also on my real workload (loading 60GB jsonl file containing mostly strings) I
measured a 10% improvement
--
___
Python tracker
<https://bugs.python.org/issue37
Marco Paolini added the comment:
Here's the real world example
$ ls -hs events-100k.json
84M events-100k.json
+---+-+-+
| Benchmark | vanilla-bpo-events-100k | patched-bpo-events-100k
Marco Paolini added the comment:
I forgot to mention, I was inspired by @christian.heimes 's talk at EuroPython
2019
https://ep2019.europython.eu/talks/es2pZ6C-introduction-to-low-level-profiling-and-tracing/
(thanks!)
--
___
Python tracker
Marco Paolini added the comment:
I am also working on a different patch that uses the "pcmpestri" SSE4 processor
instruction, it looks like this for now.
While at it I realized there is (maybe) another potential speedup: avoiding the
ucs4lib_find_max_char we do for each chunk of
Marco Paolini added the comment:
On gcc, running the tests above, the only change that is relevant for speedup
is switching around the strict check. Removing the extra MOV related to the
outer "c" variable is not significant (at least on gcc and the few tests I did)
Unfortuna
Marco Paolini added the comment:
ujson (https://github.com/esnme/ultrajson) instead is faster when decoding
non-ascii in the same example above, so it is likely there is room for
improvement...
--
___
Python tracker
<https://bugs.python.
Marco Paolini added the comment:
I also confirm Inada's patch further improves performance!
All my previous benchmarks were done with gcc and PGO optimizations performed
only with test_json task... maybe this explains the weird results?
I tested the performance of new master
Marco Paolini added the comment:
also worth noting escape sequences for non-ascii characters are slower, even
when encoded length is the same.
python -m pyperf timeit -s 'import json;' -s 'c = "€"; s = json.dumps(c *
(2**10 // len(json.dumps(c)) - 2))' 'json.loads(s)' -o nonas
Marco Paolini added the comment:
ops sorry here's the right commands
python -m pyperf timeit -s 'import json;' -s 'c = "a"; s = json.dumps(c *
(2**10 // (len(json.dumps(c)) - 2)))' 'json.loads(s)' -o ascii2k.json
python -m pyperf timeit -s 'import json;' -s 'c = "€"; s
Marco Paolini added the comment:
@steve.dower yes, that's what made me discard that experiment we did during the
sprint.
Ok will test your new patch soon
--
___
Python tracker
<https://bugs.python.org/issue37
New submission from Marco Paolini :
Module two-phase initialization does not report errors correctly to the import
machinery
--
components: Extension Modules
messages: 355913
nosy: mpaolini
priority: normal
severity: normal
status: open
title: Arraymodule initialization error handling
Change by Marco Paolini :
--
keywords: +patch
pull_requests: +16552
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/17039
___
Python tracker
<https://bugs.python.org/issu
Marco Paolini added the comment:
This happens because the default value for the start argument is zero , hence
the first operation is `0 + 'a'`
--
nosy: +mpaolini
___
Python tracker
<https://bugs.python.org/issue41
Marco Paolini added the comment:
also worth noting, the start argument is type checked instead. Maybe we could
apply the same checks to the items of the iterable?
python3 -c "print(sum(('a', 'b', 'c'), start='d'))"
Traceback (most recent call last):
File "", line 1
Marco Paolini added the comment:
I was thinking to just clarify a bit the error message that results from
Py_NumberAdd. This won't make it slower in the "hot" path
doing something like (not compile tested, sorry)
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -2451,
Marco Paolini added the comment:
hello Thomas,
do you need any help fixing the conflicts in your PR?
even if Lib/warnings.py changed a little in the last 2 years, your PR is still
good!
--
nosy: +mpaolini
___
Python tracker
<ht
36 matches
Mail list logo
