sdf-jkl opened a new pull request, #12247:
URL: https://github.com/apache/arrow-rs/pull/12247
# Which issue does this PR close?
- Closes #11627.
# Rationale for this change
`VariantPath::try_from(&str)` had a grammar of its own: paths without a root
(`a.b`), `""` for the root, and a backslash in brackets that dropped itself
before any character. It matched neither JSONPath nor Spark. It also read `$`
as a field name, so a Spark path such as `$.a` looked up a field called `$` and
returned nothing.
#11627 settled on Spark's `variant_get` path syntax. Spark versions differ
only in quoted field names: 4.2 and 4.3 keep backslashes literal, and 4.4
(SPARK-59218, on master) decodes escapes. This PR supports both and decodes by
default.
# What changes are included in this PR?
- `VariantPath::try_from(&str)` follows Spark's `VariantPathParser`: `$`,
then any number of `.name`, `['name']`, `["name"]` or `[N]`. `$` alone is the
empty path.
- New `VariantPath::parse(path, QuotedKeyEscapes)`. `Decode`, the default
used by `try_from`, decodes `\\`, `\'`, `\"`, `\/`, `\b`, `\f`, `\n`, `\r`,
`\t`, `\uXXXX` and surrogate pairs as Spark 4.4 does. `Literal` keeps
backslashes as Spark 4.2 and 4.3 do, for engines that target those versions.
- `[*]` stays as an arrow-rs extension for `ShreddedSchemaBuilder`.
`variant_get` and `Variant::get_path` already reject it.
- Doc examples, tests and the path benchmark use `$` paths.
| Path | Before | After, `Decode` | After, `Literal` |
|---|---|---|---|
| `a.b` | `a`, `b` | error | error |
| `""` | root | error | error |
| `$.a` | `$`, `a` | `a` | `a` |
| `$.a]b` | error | `a]b` | `a]b` |
| `$['a\nb']` | `$`, `anb` | `a`, newline, `b` | `a\nb` |
| `$['a\qb']` | `$`, `aqb` | `a\qb` | `a\qb` |
| `$[2147483648]` | `$`, index 2147483648 | error | error |
# Are these changes tested?
Yes. `parquet-variant/src/path.rs` has table tests for valid paths, invalid
paths, quoted escapes in both modes, error messages and borrowed field names.
One test builds an object with the keys `anb`, `a\nb` and `a<newline>b` and
checks that each mode selects a different key.
The expected values come from Spark. I ran Spark's own `VariantPathParser`
(the pyspark 4.3.0 jars, and the spark-catalyst 4.4.0-SNAPSHOT jar built
2026-11-12) and this parser on the same 91 paths. `Literal` matches 4.3 and
`Decode` matches 4.4 on every path except `$[*]`, which Spark rejects.
# Are there any significant user-facing changes?
Yes, these are breaking:
- String paths must start with `$`. Prefix `$.`, or `$` before a leading
`[`. Use `$` instead of `""` for the root. This includes
`ShreddedSchemaBuilder::with_path("$.items[*].id", ...)`.
- Backslashes in quoted field names have new meanings, shown in the table
above.
- `.name` runs to the next `.` or `[`, so it can now contain `]`. Indexes
above `i32::MAX` are errors.
New public API: `VariantPath::parse` and `QuotedKeyEscapes`.
Writing a `VariantPath` back to a string (Spark's `quoteKey`) is not part of
this PR.
# AI usage disclosure
Claude Code wrote the parser port, the tests, the caller updates and this
description, and ran the comparison against Spark.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]