xiangfu0 opened a new pull request, #18972:
URL: https://github.com/apache/pinot/pull/18972
## Summary
Ingestion and query-time JsonPath extraction goes through Jayway
(`parse(json).read(path)`), which builds a full Jackson DOM of the document
and only then walks to
the field — once per row per derived column, a major ingestion CPU cost.
This adds a streaming
Jackson-based extractor for **simple linear paths** (`$` followed only by
`.key`, `['key']`,
`[int]`), used by both re-parse call sites:
- `JsonFunctions.jsonPath` (and `jsonPathString/Long/Double/Array/Exists`,
which route through it)
- `JsonExtractScalarTransformFunction` (ingestion `transformConfigs`)
Everything else falls back to Jayway: wildcards, deep scan (`..`), filters,
unions, slices,
negative indices, non-`String`/`byte[]` input, and any document whose first
non-whitespace character
is not `{` or `[` (bare scalars, `null`, empty, whitespace, BOM, plain text).
## Feature flags — both default **OFF**
| Key | Effect |
|---|---|
| `pinot.jsonpath.fastpath.enabled` | Gate the streaming fast path. |
| `pinot.jsonpath.fastpath.earlyExit.enabled` | Also stop at the leaf
instead of scanning the whole root value. No effect unless the above is on. |
Pure kill-switch back to Jayway. Server-local compute only — no wire format,
segment format, or
config-schema change, so mixed-version safe with no migration.
## Semantics
The default (full-scan) path is **byte-for-byte identical** to Jayway as
Pinot configures it
(`JacksonJsonProvider` + `JacksonMappingProvider` + `SUPPRESS_EXCEPTIONS`),
including duplicate-key
last-wins and raising `InvalidJsonException` on a document malformed
anywhere inside its root value.
Content *after* the root value is ignored, exactly as
`ObjectMapper.readValue` ignores it.
**Early exit** trades two behaviors for speed when the field appears early,
since the tail is never
read: (1) duplicate keys resolve to the *first* occurrence; (2) a document
malformed strictly after
the addressed field no longer raises. Both inputs are RFC-8259-undefined or
corrupt, and the switch
is off by default.
## Correctness gate
`StreamingJsonPathExtractorTest` is a **differential test with the
production Jayway fallback as the
oracle**: the same `jsonPath*` entry points run flag-off vs flag-on and must
agree, over a
hand-picked matrix (missing paths, invalid JSON, null/empty/whitespace/BOM,
type mismatches, nested
arrays, arrays of objects, deep nesting, unicode/surrogates,
escaped/dotted/duplicate keys,
big/precise numbers) **plus 60k randomized fuzz iterations**, the `byte[]`
overloads (incl. multibyte
and invalid-UTF-8), and the `USE_BIG_DECIMAL_FOR_FLOATS` context. Early-exit
divergences are
asserted explicitly. `JsonFunctionsFastPathTest` and
`JsonExtractScalarTransformFunctionFastPathTest`
re-run the existing corpora through the fast path.
## Performance
JMH (`BenchmarkJsonPathExtraction`, Apple M, ~688-byte nested payload, 4×5s
warmup / 6×5s measure);
the `fastPathEnabled=false` arm is the current Jayway baseline, measured in
the same run:
| `jsonPathString`, one column | ops/s | vs Jayway |
|---|---:|---:|
| Jayway (today) | 458k | 1.0x |
| Streaming, **exact parity** (full scan) | 836k | **1.8x** |
| Streaming, early exit, field early | 5.09M | **11.1x** |
| Streaming, early exit, field last | 813k | 1.8x |
| 4 derived columns from one document | ops/s | vs Jayway |
|---|---:|---:|
| Jayway (4 parses) | 106k | 1.0x |
| Streaming, one pass, all 4 paths | 763k | **7.2x** |
1.8x is the Jackson lexing floor: exact parity requires reading the whole
root value, so a bare
`nextToken()`-to-EOF loop (~907 ns) is already close to the full extract
(~919 ns). The larger wins
come from early exit (undefined-input tradeoff) and from single-pass
multi-path extraction, which
must walk the document anyway and so gets exact parity for free — the right
target for
`transformConfigs` pulling N columns from one source.
## Testing
`pinot-common`: 575 tests green (incl. the differential + fuzz).
`pinot-core`: 218 green
(`JsonExtractScalarTransformFunctionTest` + its fast-path subclass,
`JsonExtractScalarTest`,
`JsonExtractIndexTransformFunctionTest`). spotless / checkstyle / license
clean.
## Notes for reviewers
- `earlyExit` changes results, so it must be flipped **cluster-wide, not
mid-rolling-restart** — a
half-restarted cluster would have servers disagree within one
scatter-gather. `enabled` alone is
result-preserving and rolls normally.
- The single-pass multi-path `extract` is included and benchmarked but not
yet wired into
`jsonExtractScalar` (which would need sibling transform functions to share
one pass over the source
column); that is the follow-up where the 7x lands in ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
