andygrove opened a new issue, #4947:
URL: https://github.com/apache/datafusion-comet/issues/4947
## Describe the bug
`get_json_object` diverges from Spark when a JSON object contains duplicate
keys.
For input `{"a":1,"a":2}` and path `$.a`:
- Comet returns `2` (the last occurrence)
- Spark returns `1` (the first occurrence)
Spark's `GetJsonObjectEvaluator.evaluatePath`
(`sql/catalyst/.../json/JsonExpressionEvalUtils.scala`) stops at the first
matching field once `dirty` is set, so it returns the first occurrence. Comet
visits every entry and resolves to the last occurrence, matching `serde_json`'s
`preserve_order` overwrite semantics rather than Spark's.
This is a pre-existing divergence, not a regression. It predates the
streaming rewrite in #4907 (the old `serde_json` + `preserve_order` path had
the same last-wins behavior). It was noted during review of #4907.
## Steps to reproduce
```sql
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2
```
## Expected behavior
Match Spark and return the first occurrence of a duplicated key.
## Additional context
The fix is on the Rust side in
`native/spark-expr/src/string_funcs/get_json_object.rs`:
`SegmentVisitor::visit_map` should stop updating `found` once a key has matched
(while still consuming the remaining entries to validate the document). The
`test_duplicate_key_last_wins` test added in #4907 would need to be flipped to
first-wins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
