mbutrovich opened a new issue, #3718:
URL: https://github.com/apache/iceberg-rust/issues/3718

   ### Is your feature request related to a problem or challenge?
   
   iceberg-rust can't read expressions in the JSON form that [Appendix B of the 
expressions 
spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L231-L297)
 defines. Three readers need it:
   
   - `pyiceberg-core` needs it to receive scan filters from PyIceberg for 
#3649, because PyIceberg serializes its expressions to that JSON.
   - The REST scan planning client (#1690) needs it to read the 
`residual-filter` of each scan task.
   - v4 table metadata stores expressions in this form in three places: the 
`expression` of a `check` constraint 
([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L779-L783)),
 a `write-default` that is a value expression 
([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L379-L385)),
 and the `expr` of an `expr-value` stats field 
([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L507)).
 Check constraints and `expr-value` stats use ID references, and both can call 
functions with `apply`.
   
   A `Predicate` can't hold most of what v4 metadata stores. Its terms are 
named references only, so it has no value expressions, no `apply`, and no ID 
references. Metadata also has to write back expressions that iceberg-rust can't 
evaluate without losing them.
   
   ### Describe the solution you'd like
   
   Parse in two layers, both in `iceberg::expr`.
   
   The first layer is a set of `pub(crate)` types that mirror the Appendix B 
grammar. They cover value expressions (literals with an optional `data-type`, 
references by `id` or `name`, and `apply` with its function reference and 
arguments) and predicates. They derive serde, read both the spec form and the 
[deprecated term-based 
form](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L185-L197),
 and write the spec form. They don't need a schema, and they keep every 
expression they read, so writing it back loses nothing. Literal values stay as 
JSON in this layer, because their type isn't known without a schema.
   
   The second layer is `Predicate::try_from_json(value: serde_json::Value, 
schema: &Schema, case_sensitive: bool) -> Result<Predicate>`. The name and the 
`serde_json::Value` argument follow 
[`Literal::try_from_json`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/values/literal.rs#L408),
 and REST types that hold a `serde_json::Value` can pass it straight through. 
It reads the JSON into the first layer's types, then converts them:
   
   - It types each literal from the field it's compared with, using 
`Literal::try_from_json`. Literals are untyped [JSON single 
values](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L2248),
 so a date arrives as `"2007-12-03"`. Java does the same with 
`SingleValueParser` when it's given a schema 
([`ExpressionParser.java#L326`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L326)).
 The conversion can't wait until binding, because 
[`Datum::to`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/values/datum.rs#L1195-L1217)
 doesn't convert strings to date, decimal, uuid, fixed, or binary.
   - It resolves ID references to names through the schema. A `Predicate` is 
evaluated against the current schema, so the name is enough there. The first 
layer keeps the ID for metadata.
   - It returns an error for anything a `Predicate` can't hold: `apply`, 
deprecated `transform` terms, and comparisons that don't put a reference on the 
left and a literal on the right. It also returns an error for null and NaN 
literals, which the [spec 
rejects](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L159).
   - `case_sensitive` applies to name lookups, as in `Predicate::bind`.
   
   It returns an unbound `Predicate`, which is what 
[`TableScanBuilder::with_filter`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L212)
 takes, so each caller binds it with its own schema and case sensitivity.
   
   Both layers go in `iceberg::expr` because that module owns the expression 
types. `spec` already imports from `expr` 
([`spec/transform.rs#L32`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/transform.rs#L32)),
 so v4 metadata can hold the first layer's types. Java does the same: 
apache/iceberg#17343 adds its new expression types next to the rest of the 
expression model. The code goes in the `iceberg` crate and not in 
`iceberg-catalog-rest`, because the JSON form comes from a format spec that 
metadata uses too, not only the REST protocol. `pyiceberg-core` depends on 
`iceberg` but not on `iceberg-catalog-rest`, and `iceberg-catalog-rest` depends 
on `iceberg`, so both can reach it there. Java keeps `ExpressionParser` in 
`core`, and its REST parsers call it 
([`RESTFileScanTaskParser.java#L102`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/rest/RESTFileS
 canTaskParser.java#L102), 
[`PlanTableScanRequestParser.java#L118`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java#L118)).
   
   The second layer is a function and not a `Deserialize` impl on `Predicate`, 
for two reasons. Typing literals needs the schema, which `Deserialize` can't 
take. And `Predicate` and `BoundPredicate` already derive serde with a 
different shape, which serialized `FileScanTask`s use for their predicate 
([`task.rs#L478`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/task.rs#L478)).
 Changing the derive would break them.
   
   This adds one public item, `Predicate::try_from_json`. The first layer stays 
`pub(crate)` because iceberg-rust has no v4 support yet, and #3513 collects v4 
prototypes that aren't ready to merge. Making the types public when v4 metadata 
exposes them is additive. Making them public now would fix their shape before 
any v4 code uses them.
   
   Accepting both forms matters today. PyIceberg emits the deprecated form 
([`expressions/__init__.py#L895`](https://github.com/apache/iceberg-python/blob/9955502c73758611d5ad69220b67aedb3d4a443e/pyiceberg/expressions/__init__.py#L895)).
 Java's `ExpressionParser` reads only the deprecated form 
([`ExpressionParser.java#L321`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L321),
 
[`#L409-L411`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L409-L411)),
 and apache/iceberg#17343 adds the spec form to Java.
   
   First-layer tests read every grammar production in both forms and check that 
each writes back in the spec form unchanged. That includes `apply` with each 
function reference form, ID and named references, and the `{"type": "literal"}` 
and `{"type": "literals"}` object forms with and without `data-type`. 
Second-layer tests cover every operator, bare `true` and `false`, a literal of 
every primitive type, nested field names, case sensitivity, unknown fields and 
IDs, and each error case.
   
   Writing a `Predicate` as JSON is <writer>.
   
   ### Willingness to contribute
   
   I can contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to