sdf-jkl opened a new pull request, #12247:
URL: https://github.com/apache/arrow-rs/pull/12247

   # Which issue does this PR close?
   
   - Closes #11627.
   
   # Rationale for this change
   
   `VariantPath::try_from(&str)` had a grammar of its own: paths without a root 
(`a.b`), `""` for the root, and a backslash in brackets that dropped itself 
before any character. It matched neither JSONPath nor Spark. It also read `$` 
as a field name, so a Spark path such as `$.a` looked up a field called `$` and 
returned nothing.
   
   #11627 settled on Spark's `variant_get` path syntax. Spark versions differ 
only in quoted field names: 4.2 and 4.3 keep backslashes literal, and 4.4 
(SPARK-59218, on master) decodes escapes. This PR supports both and decodes by 
default.
   
   # What changes are included in this PR?
   
   - `VariantPath::try_from(&str)` follows Spark's `VariantPathParser`: `$`, 
then any number of `.name`, `['name']`, `["name"]` or `[N]`. `$` alone is the 
empty path.
   - New `VariantPath::parse(path, QuotedKeyEscapes)`. `Decode`, the default 
used by `try_from`, decodes `\\`, `\'`, `\"`, `\/`, `\b`, `\f`, `\n`, `\r`, 
`\t`, `\uXXXX` and surrogate pairs as Spark 4.4 does. `Literal` keeps 
backslashes as Spark 4.2 and 4.3 do, for engines that target those versions.
   - `[*]` stays as an arrow-rs extension for `ShreddedSchemaBuilder`. 
`variant_get` and `Variant::get_path` already reject it.
   - Doc examples, tests and the path benchmark use `$` paths.
   
   | Path | Before | After, `Decode` | After, `Literal` |
   |---|---|---|---|
   | `a.b` | `a`, `b` | error | error |
   | `""` | root | error | error |
   | `$.a` | `$`, `a` | `a` | `a` |
   | `$.a]b` | error | `a]b` | `a]b` |
   | `$['a\nb']` | `$`, `anb` | `a`, newline, `b` | `a\nb` |
   | `$['a\qb']` | `$`, `aqb` | `a\qb` | `a\qb` |
   | `$[2147483648]` | `$`, index 2147483648 | error | error |
   
   # Are these changes tested?
   
   Yes. `parquet-variant/src/path.rs` has table tests for valid paths, invalid 
paths, quoted escapes in both modes, error messages and borrowed field names. 
One test builds an object with the keys `anb`, `a\nb` and `a<newline>b` and 
checks that each mode selects a different key.
   
   The expected values come from Spark. I ran Spark's own `VariantPathParser` 
(the pyspark 4.3.0 jars, and the spark-catalyst 4.4.0-SNAPSHOT jar built 
2026-11-12) and this parser on the same 91 paths. `Literal` matches 4.3 and 
`Decode` matches 4.4 on every path except `$[*]`, which Spark rejects.
   
   # Are there any significant user-facing changes?
   
   Yes, these are breaking:
   
   - String paths must start with `$`. Prefix `$.`, or `$` before a leading 
`[`. Use `$` instead of `""` for the root. This includes 
`ShreddedSchemaBuilder::with_path("$.items[*].id", ...)`.
   - Backslashes in quoted field names have new meanings, shown in the table 
above.
   - `.name` runs to the next `.` or `[`, so it can now contain `]`. Indexes 
above `i32::MAX` are errors.
   
   New public API: `VariantPath::parse` and `QuotedKeyEscapes`.
   
   Writing a `VariantPath` back to a string (Spark's `quoteKey`) is not part of 
this PR.
   
   # AI usage disclosure
   
   Claude Code wrote the parser port, the tests, the caller updates and this 
description, and ran the comparison against Spark.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to