[issue23493] optimize sort_keys in json module by using operator.itemgetter()

STINNER Victor added the comment: > I am wondering why the change was not backported to 3.6 and 3.7? These branches no longer accept optimizations or bugfixes, only security fixes: https://devguide.python.org/#status-of-python-branches -- ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Mickaël Schoentgen added the comment: I am wondering why the change was not backported to 3.6 and 3.7? It introduces different behavior. For instance, I need to keep duplicate keys from JSON data (because it is allowed by the RFC and it is a missing feature for tools such like HTTPie). Have

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

INADA Naoki added the comment: New changeset e25399b40cd15620e77c9ad2ed24549006ae9b47 by INADA Naoki in branch 'master': bpo-23493: json: Change sort_keys in Python encoder same to C (GH-8131) https://github.com/python/cpython/commit/e25399b40cd15620e77c9ad2ed24549006ae9b47 --

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

INADA Naoki added the comment: I can't confirm significant performance benefit. sort_keys is not bottleneck of pure Python encoder. BTW, benefit from C encoder is more significant. It's about 3x faster. Adding `indent` option support to C encoder may be good target for new contributors who

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Change by INADA Naoki : -- keywords: +patch pull_requests: +7709 stage: -> patch review ___ Python tracker ___ ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

INADA Naoki added the comment: In C version, we don't use key func. items = PyMapping_Items(dct); if (items == NULL) goto bail; if (s->sort_keys && PyList_Sort(items) < 0) { Like that, how about removing key function completely? -- nosy: +inada.naoki

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23493 ___ ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Serhiy Storchaka added the comment: May be there is a time to optimize creating Python frames (at least for the case when one frame is created and destroyed multiple times). May be frame pool? Or cached one frame per function? -- ___ Python tracker

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Serhiy Storchaka added the comment: Unless we use recent PyPy with ordered dicts. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23493 ___ ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

STINNER Victor added the comment: Oh, forget my comment. sorted() never changes the input list, so the microbenchmark is ok. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23493 ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Serhiy Storchaka added the comment: My interpretation of the results. In CPython using operator.itemgetter() makes sorting up to 2 times faster (only 25% faster with string keys), in PyPy it make sorting slightly slower (because operator.itemgetter() is implemented in Python, but the lambda

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Serhiy Storchaka added the comment: CPython: $ python3.4 -m timeit -s 'f = lambda kv: kv[0]' -s 's = list(dict.fromkeys(range(1000)).items())' -- 'sorted(s, key=f)' 1000 loops, best of 3: 904 usec per loop $ python3.4 -m timeit -s 'import operator' -s 'f = operator.itemgetter(0)' -s 's =

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

STINNER Victor added the comment: If I remember correctly, the complexity and performance of sort/sorted depends if the data set is sorted or not. You may recreated the list/dictionary at each iteration to get performances closer to items = sorted(dct.items(), key=lambda kv: kv[0]) (dict keys

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

STINNER Victor added the comment: That said, it is applied only n-times and is likely insignificant when compared to the O(n log n) sort. (...) 904 usec = 462 usec is very significant: it's 49% faster. So I'm ok for the change. Note: PyPy JIT may not be able to optimize

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

STINNER Victor added the comment: My interpretation of the results. In CPython using operator.itemgetter() makes sorting up to 2 times faster If I remember correctly, Python functions implemented in C don't create a Python frame. The Python frame is an important cost in term of performances.

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Changes by Serhiy Storchaka storch...@gmail.com: -- status: languishing - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23493 ___ ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Serhiy Storchaka added the comment: Could you measure the effect on json.encode()? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23493 ___ ___

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Wouter Bolsterlee added the comment: Using IPython and CPython 3.4: d = dict.fromkeys(map(str, range(1000))) Current implementation: %timeit sorted(d.items(), key=lambda kv: kv[0]) 1000 loops, best of 3: 605 µs per loop %timeit sorted(d.items(), key=lambda kv: kv[0]) 1000 loops, best of

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Raymond Hettinger added the comment: I'm -0 on this proposal. There is a small speed-up to be had here: $ python3.4 -m timeit -s 'f=lambda kv: kv[0]' -s 's=[10, 20]' 'f(s)' 1000 loops, best of 3: 0.105 usec per loop $ python3.4 -m timeit -s 'import operator' -s

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

Josh Rosenberg added the comment: Is it even legal to have non-string keys in a JSON object? If they must be strings, and they must be unique, I don't think a key argument is necessary (and it would save the generation of the key array; not doing the work is faster than doing the work more

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

New submission from Wouter Bolsterlee: The JSON encoder uses a lambda function for the sort(key=...) invocation used to sort the keys in a JSON object in case sort_keys=True is passed: https://hg.python.org/cpython/file/46bfddb14cbe/Lib/json/encoder.py#l352 Instead of having a lambda,