New submission from Marco Paolini <markopaol...@gmail.com>:
I analysed the performance of json.loads in one production workload we have.
Spy-py tells me the majority of time is spent into C json module (see
events.svg)
Digging deeper, linux perf tells me hottest loop (where 20%+ of the time is
spent) in _json.scanstring_unicode, in this loop:
189: movzx eax,BYTE PTR [rbp+rbx*1+0x0]
mov DWORD PTR [rsp+0x44],eax
cmp eax,0x22
je 22f
cmp eax,0x5c
je 22f
test r13d,r13d
je 180
cmp eax,0x1f
which is related to this code in Modules/_json.c
/* Find the end of the string or the next escape */
Py_UCS4 c = 0;
for (next = end; next < len; next++) {
c = PyUnicode_READ(kind, buf, next);
if (c == '"' || c == '\\') {
break;
}
else if (strict && c <= 0x1f) {
raise_errmsg("Invalid control character at", pystr, next);
goto bail;
}
}
Two optimisations can be done:
1. remove the mov entirely. It is not needed inside the loop and it is only
needed later, outside the loop to access the variable
2. switch around the strict check (in the second if) because strict defaults to
1 so it will likely pass the test, while the likelyness of finding an invalid
character is lower
Running the pyperformance json_loads benchmark I get:
+------------+----------------------+-----------------------------+
| Benchmark | vanilla-pyperf-pgo38 | patched-pyperf-pgo38 |
+============+======================+=============================+
| json_loads | 54.9 us | 53.9 us: 1.02x faster (-2%) |
+------------+----------------------+-----------------------------+
A micro benchmark on a 1MB long json string gives better results:
python -m pyperf timeit -s "import json; x = json.dumps({'k': '1' * 2 ** 20})"
"json.loads(x)"
+-----------+------------+-----------------------------+
| Benchmark | vanilla-1m | patched-1m |
+===========+============+=============================+
| timeit | 2.62 ms | 2.39 ms: 1.10x faster (-9%) |
+-----------+------------+-----------------------------+
----------
components: Library (Lib)
files: events.svg
messages: 347832
nosy: christian.heimes, mpaolini, pablogsal
priority: normal
severity: normal
status: open
title: JSON loads performance improvement for long strings
type: performance
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file48476/events.svg
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue37587>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
