sdf-jkl opened a new issue, #11627:
URL: https://github.com/apache/arrow-rs/issues/11627
### Is your feature request related to a problem or challenge?
We've referred to both Spark/Databricks paths and JSONPath while building
VariantPath, but haven't settled which behavior we're targeting when they
differ.
Escaping is one example: Arrow reads `['a\nb']` as the key `anb`. Spark 4.2
and 4.3 keep the backslash and read `a\nb`. Spark master, targeting 4.4, reads
`\n` as a newline (apache/spark#59097, merged as SPARK-59218), and
`spark.sql.legacy.variantPathBackslashAsLiteral` restores the older behavior.
Arrow also accepts `\]`, which RFC 9535 rejects.
### Describe the solution you'd like
Agree on the target and document the supported subset:
- Should VariantPath follow Spark/Databricks, RFC 9535, or its existing
behavior?
- How should we handle accepted paths whose meaning would change?
- Which extensions belong to specific consumers, such as `[*]` for shredding
schemas versus singular paths for `variant_get`?
Capture the decision in a small compatibility table and tests, starting with
quotes and backslash escapes.
### Describe alternatives you've considered
Keep the current grammar and document its differences. That's also a valid
choice; the goal is to make future parsing decisions consistent.
### Additional context
- [#9012
discussion](https://github.com/apache/arrow-rs/pull/9012#discussion_r2640273680)
proposed a Databricks-compatible subset.
- #9276 introduced arbitrary backslash escaping for unquoted bracket fields;
#9479 later required quotes while retaining that behavior.
- #11231 fixed bracket handling inside quotes and preserved `\]` for
compatibility.
- datafusion-contrib/datafusion-variant#143 parses rooted paths per Spark
4.2. datafusion-contrib/datafusion-variant#166 tracks dropping its rootless
paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]