mbutrovich opened a new issue, #3718: URL: https://github.com/apache/iceberg-rust/issues/3718
### Is your feature request related to a problem or challenge? iceberg-rust can't read expressions in the JSON form that [Appendix B of the expressions spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L231-L297) defines. Three readers need it: - `pyiceberg-core` needs it to receive scan filters from PyIceberg for #3649, because PyIceberg serializes its expressions to that JSON. - The REST scan planning client (#1690) needs it to read the `residual-filter` of each scan task. - v4 table metadata stores expressions in this form in three places: the `expression` of a `check` constraint ([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L779-L783)), a `write-default` that is a value expression ([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L379-L385)), and the `expr` of an `expr-value` stats field ([spec](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L507)). Check constraints and `expr-value` stats use ID references, and both can call functions with `apply`. A `Predicate` can't hold most of what v4 metadata stores. Its terms are named references only, so it has no value expressions, no `apply`, and no ID references. Metadata also has to write back expressions that iceberg-rust can't evaluate without losing them. ### Describe the solution you'd like Parse in two layers, both in `iceberg::expr`. The first layer is a set of `pub(crate)` types that mirror the Appendix B grammar. They cover value expressions (literals with an optional `data-type`, references by `id` or `name`, and `apply` with its function reference and arguments) and predicates. They derive serde, read both the spec form and the [deprecated term-based form](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L185-L197), and write the spec form. They don't need a schema, and they keep every expression they read, so writing it back loses nothing. Literal values stay as JSON in this layer, because their type isn't known without a schema. The second layer is `Predicate::try_from_json(value: serde_json::Value, schema: &Schema, case_sensitive: bool) -> Result<Predicate>`. The name and the `serde_json::Value` argument follow [`Literal::try_from_json`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/values/literal.rs#L408), and REST types that hold a `serde_json::Value` can pass it straight through. It reads the JSON into the first layer's types, then converts them: - It types each literal from the field it's compared with, using `Literal::try_from_json`. Literals are untyped [JSON single values](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/spec.md?plain=1#L2248), so a date arrives as `"2007-12-03"`. Java does the same with `SingleValueParser` when it's given a schema ([`ExpressionParser.java#L326`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L326)). The conversion can't wait until binding, because [`Datum::to`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/values/datum.rs#L1195-L1217) doesn't convert strings to date, decimal, uuid, fixed, or binary. - It resolves ID references to names through the schema. A `Predicate` is evaluated against the current schema, so the name is enough there. The first layer keeps the ID for metadata. - It returns an error for anything a `Predicate` can't hold: `apply`, deprecated `transform` terms, and comparisons that don't put a reference on the left and a literal on the right. It also returns an error for null and NaN literals, which the [spec rejects](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/format/expressions-spec.md?plain=1#L159). - `case_sensitive` applies to name lookups, as in `Predicate::bind`. It returns an unbound `Predicate`, which is what [`TableScanBuilder::with_filter`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/mod.rs#L212) takes, so each caller binds it with its own schema and case sensitivity. Both layers go in `iceberg::expr` because that module owns the expression types. `spec` already imports from `expr` ([`spec/transform.rs#L32`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/transform.rs#L32)), so v4 metadata can hold the first layer's types. Java does the same: apache/iceberg#17343 adds its new expression types next to the rest of the expression model. The code goes in the `iceberg` crate and not in `iceberg-catalog-rest`, because the JSON form comes from a format spec that metadata uses too, not only the REST protocol. `pyiceberg-core` depends on `iceberg` but not on `iceberg-catalog-rest`, and `iceberg-catalog-rest` depends on `iceberg`, so both can reach it there. Java keeps `ExpressionParser` in `core`, and its REST parsers call it ([`RESTFileScanTaskParser.java#L102`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/rest/RESTFileS canTaskParser.java#L102), [`PlanTableScanRequestParser.java#L118`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/rest/requests/PlanTableScanRequestParser.java#L118)). The second layer is a function and not a `Deserialize` impl on `Predicate`, for two reasons. Typing literals needs the schema, which `Deserialize` can't take. And `Predicate` and `BoundPredicate` already derive serde with a different shape, which serialized `FileScanTask`s use for their predicate ([`task.rs#L478`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/scan/task.rs#L478)). Changing the derive would break them. This adds one public item, `Predicate::try_from_json`. The first layer stays `pub(crate)` because iceberg-rust has no v4 support yet, and #3513 collects v4 prototypes that aren't ready to merge. Making the types public when v4 metadata exposes them is additive. Making them public now would fix their shape before any v4 code uses them. Accepting both forms matters today. PyIceberg emits the deprecated form ([`expressions/__init__.py#L895`](https://github.com/apache/iceberg-python/blob/9955502c73758611d5ad69220b67aedb3d4a443e/pyiceberg/expressions/__init__.py#L895)). Java's `ExpressionParser` reads only the deprecated form ([`ExpressionParser.java#L321`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L321), [`#L409-L411`](https://github.com/apache/iceberg/blob/13135ae08148b010276fc9c5eb1cfbf2e6e8289c/core/src/main/java/org/apache/iceberg/expressions/ExpressionParser.java#L409-L411)), and apache/iceberg#17343 adds the spec form to Java. First-layer tests read every grammar production in both forms and check that each writes back in the spec form unchanged. That includes `apply` with each function reference form, ID and named references, and the `{"type": "literal"}` and `{"type": "literals"}` object forms with and without `data-type`. Second-layer tests cover every operator, bare `true` and `false`, a literal of every primitive type, nested field names, case sensitivity, unknown fields and IDs, and each error case. Writing a `Predicate` as JSON is <writer>. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
