u70b3 opened a new pull request, #4971:
URL: https://github.com/apache/datafusion-comet/pull/4971
## Which issue does this PR close?
Closes #4947.
## Rationale for this change
For a JSON object with duplicate keys, Spark's `GetJsonObjectEvaluator`
stops at the first matching field, but Comet's native implementation resolved
the key to its *last* occurrence (matching `serde_json`'s overwrite semantics
rather than Spark's):
```sql
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet (before this PR): 2
```
## What changes are included in this PR?
`SegmentVisitor::visit_map` in
`native/spark-expr/src/string_funcs/get_json_object.rs` now locks in the first
key match and consumes later occurrences via `IgnoredAny` instead of re-parsing
them with `PathSeed`:
- The lock is keyed on the *key match*, not on a successful subpath
resolution: if the first matching value does not contain the rest of the path,
the result is `null` and the second duplicate key is never consulted, matching
Spark.
- All entries are still visited, so the existing malformed/trailing-garbage
rejection behavior is unchanged.
## How are these changes tested?
- Flipped the existing `test_duplicate_key_last_wins` unit test to
`test_duplicate_key_first_wins`.
- Added `test_duplicate_key_first_wins_nested` (`{"a":{"b":1},"a":{"b":2}}`
/ `$.a.b` -> `1`).
- Added `test_duplicate_key_first_match_locks_value`
(`{"a":{"x":1},"a":{"b":2}}` / `$.a.b` -> `null`, matching Spark's
lock-on-first-match semantics).
- Full `datafusion-comet-spark-expr` suite passes (497 tests), plus `clippy`
and `fmt` checks.
- Checked `spark/src/test/resources/sql-tests/` and `CometJsonJvmSuite` for
duplicate-key cases: the only repeated keys there are across *different objects
inside arrays*, which are unaffected by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
