lemire opened a new pull request, #70789:
URL: https://github.com/apache/doris/pull/70789

   ### Background
   
   simdjson is a JSON parser built from source under `thirdparty/` and linked 
into the BE. The BE uses its On-Demand API to load JSON files and streams 
(`format_v2/json/json_reader.cpp`), to convert JSON text to JSONB 
(`util/jsonb_parser_simd.h`) and in the JSON functions 
(`exprs/json_functions.cpp`), and its DOM API to parse JSON into VARIANT 
(`util/json/simd_json_parser.h`). At startup the BE selects simdjson's AVX2 
(`haswell`) kernel (`init_simdjson_parser`).
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Release/Dependency
   
   Problem Summary:
   
   Doris pins simdjson 3.11.6. simdjson 5.2.0 parses faster, mostly when 
walking documents. With the kernel Doris uses, BE-level measurements show JSON 
load 1 to 12% faster, JSONB conversion 6 to 18% faster and VARIANT parsing 3 to 
4% faster; one JSON path lookup is 12% slower (see Notes for Reviewers). This 
PR upgrades simdjson to 5.2.0, ports Doris's `BOOL` to `BOOLEAN` patch, and 
adds the one `default` case the new version needs.
   
   ### Details
   
   - **Versions:** simdjson 3.11.6 → 5.2.0, from the GitHub tag tarball, with 
`SIMDJSON_MD5SUM` updated in `thirdparty/vars.sh`. Doris renames 
`dom::element_type::BOOL` to `BOOLEAN` in its own patch, so 
`simdjson-3.0.1.patch` is replaced by `simdjson-5.2.0.patch`, the same rename 
on the 5.2.0 sources, and `download-thirdparty.sh` applies it. The build flags 
in `build-thirdparty.sh` are unchanged. `dist/LICENSE-dist.txt` listed simdjson 
3.0.1 and now lists 5.2.0.
   - **Upstream changes:** The release notes are at 
https://github.com/simdjson/simdjson/releases. The changes that matter to Doris:
     - `ondemand::json_type` gains an `unknown` value, so the top-level 
`switch` in `JsonbParser::parse` needs a `default` case to build with 
`-Werror`. It returns `InvalidArgument`.
     - Since 5.0, an integer in [2^64, 10^20) is reported as `BIGINT_ERROR` 
instead of `NUMBER_ERROR`. `JsonbParser` already handles both through 
`parse_number_success`, and no other BE code checks these codes.
     - Faster value copies in On-Demand (simdjson#3061), faster stage 1, and 
fixes in APIs that Doris does not use.
     - The license is still Apache-2.0. 5.2.0 also offers MIT and adds 
`internal/fast_float.h` (Apache-2.0 OR MIT OR BSL-1.0).
   - **Compatibility:** Built and tested on Linux x86_64 with the 
`apache/doris:build-env-ldb-toolchain-latest` toolchain (clang 20.1.7), in 
RELEASE and ASAN unit-test builds. ARM64, the GCC toolchain and macOS were not 
built. No stored data changes: simdjson only parses input. The BE build does 
not track third-party headers: an existing build directory must be rebuilt from 
scratch after the third-party update, or objects compiled against the old 
headers crash when linked with the new library.
   - **Rollback:** Revert this PR, rebuild the third-party libraries and do a 
clean BE build. No data is affected.
   
   ### Testing
   
   - `simdjson-5.2.0.patch` applies cleanly to the 5.2.0 tarball, and the 
patched library was built with the flags of `build_simdjson` into a copy of the 
`build-env-ldb-toolchain-latest` image.
   - RELEASE unit-test build (`BUILD_TYPE_UT=RELEASE ./run-be-ut.sh`), filter 
`*Json*:*JSON*:*Variant*:*VARIANT*:*Serde*:*SerDe*:*Jsonb*`: 1,486 tests, 1,471 
pass with 3.11.6 and the same 1,471 with 5.2.0. The same 11 fail with both 
versions (`ColumnVariantV2Test` invariants, two 
`DataTypeSerDeDecodedValuesDeathTest` cases and 
`QuantileStateSerdeTest.writeColumnToPb`), when the saved binaries are run 
outside `run-be-ut.sh`.
   - ASAN unit-test build from scratch with 5.2.0 (`./run-be-ut.sh --clean 
--run`, same filter): 1,483 of 1,488 tests pass, 5 are skipped, none fail, and 
no sanitizer errors are reported.
   - The prebuilt third-party packages and the build-env image still contain 
3.11.6 and need to be regenerated.
   
   ### Notes for Reviewers
   
   The speedups in Problem Summary come from a local performance test, not part 
of this PR. It was built with each simdjson version (RELEASE unit-test 
binaries, same toolchain) and the two binaries were run alternately on an Intel 
Xeon Gold 6548N, pinned to one core. Each number is the median of 3 runs, each 
run the fastest of 7 or 9 repetitions; runs differed by about 1 to 2%. Rows are 
log-like JSON objects of about 160 B, 270 B, 1.3 KB and 4.6 KB. The JSON reader 
cases read a 200,000-row file (40,000 rows for 1.3 KB) into 7 columns, one of 
them JSONB, or into 2 columns.
   
   | Workload | 3.11.6 | 5.2.0 | Speedup, `haswell` | Speedup, `icelake` |
   |---|---:|---:|---:|---:|
   | JSON reader, lines, 270 B, 7 columns | 1172 ns/row | 1130 ns/row | 1.04x | 
1.06x |
   | JSON reader, lines, 270 B, 2 columns | 513 ns/row | 507 ns/row | 1.01x | 
1.06x |
   | JSON reader, lines, 1.3 KB, 7 columns | 5364 ns/row | 4792 ns/row | 1.12x 
| 1.15x |
   | JSON reader, outer array, 270 B, 7 columns | 1134 ns/row | 1059 ns/row | 
1.07x | 1.09x |
   | `JsonBinaryValue::from_json_string`, 160 B | 308 ns | 291 ns | 1.06x | 
1.16x |
   | `JsonBinaryValue::from_json_string`, 270 B | 748 ns | 676 ns | 1.11x | 
1.17x |
   | `JsonBinaryValue::from_json_string`, 4.6 KB | 16.8 us | 14.3 us | 1.18x | 
1.20x |
   | `JsonFunctions::extract_from_object`, `$.msg`, 270 B | 85 ns | 97 ns | 
0.88x | 1.03x |
   | `JsonFunctions::extract_from_object`, missing key, 270 B | 272 ns | 276 ns 
| 0.99x | 1.01x |
   | `JsonStringToVariantEncoder`, 160 B | 585 ns | 562 ns | 1.04x | 1.05x |
   | `JsonStringToVariantEncoder`, 270 B | 1500 ns | 1451 ns | 1.03x | 1.04x |
   | `JsonStringToVariantEncoder`, 1.3 KB | 9722 ns | 9305 ns | 1.04x | 1.07x |
   
   The time columns are for `haswell`, the kernel the BE selects. The `$.msg` 
lookup is 12% slower with `haswell` in the BE binary. The same steps in a 
standalone program (`padded_string`, `iterate`, `find_field_unordered("msg")`, 
same compiler and flags) are faster with 5.2.0, 118 ns to 105 ns with `haswell` 
and 95 ns to 79 ns with `icelake`, so the slowdown is probably code placement 
in the BE binary rather than more work in simdjson.
   
   ### Release note
   
   Upgrade simdjson to 5.2.0, which speeds up JSON loading, JSONB conversion 
and VARIANT parsing.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Partially verified: built and tested on Linux 
x86_64 with clang only; ARM64 and GCC builds not run, and the prebuilt 
third-party packages are not regenerated.
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5)
   - Generated: code, the patch, and this description
   - Human review: all code reviewed by Daniel
   - Prompts: The author asked to work on a pull request that updates Doris to 
the latest simdjson and to quantify the performance benefits.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to