shiva-imc opened a new pull request, #17906: URL: https://github.com/apache/iceberg/pull/17906
## What Adds `CONTAINS` / `NOT_CONTAINS` (substring match on string terms) to the expression API, alongside the existing `STARTS_WITH` / `NOT_STARTS_WITH`. ## Why Spark emits a DSv2 `CONTAINS` predicate for the `contains()` function, but Iceberg has no matching operation, so the predicate is dropped at `SparkV2Filters` and any operation that could otherwise be answered from metadata falls back to row-level scans. The motivating case: `DELETE FROM t WHERE contains(part_col, 'x')` on an identity-partitioned column is decidable exactly from partition values, but today plans as a full copy-on-write scan-and-rewrite. With this change it plans as a metadata-only delete. ## Semantics - **Exact evaluation wherever complete values exist**: row data (`Evaluator`, `BoundLiteralPredicate`), residuals (`ResidualEvaluator`), and partition tuples — identity transforms project the predicate through unchanged, so partition pruning and `canDeleteWhere` metadata deletes decide contains exactly. - **Conservative answers from truncated bounds**: file metrics, manifest partition summaries, and metadata tables return rows-might-match (inclusive) / rows-might-not-match (strict), because substring membership is undecidable from truncated lower/upper bounds. This mirrors `STARTS_WITH`, which prunes on bounds only because prefixes survive truncation. - `NOT contains()` reaches `NOT_CONTAINS` via the existing NOT unwrap + `Operation.negate()` path; there is intentionally no `NOT_CONTAINS` entry in `SparkV2Filters` since Spark never emits it as a named predicate (same as `NOT_STARTS_WITH`). - `ExpressionParser` serialization works via the generic op-name derivation (`contains` / `not-contains`); binding validates the term produces a string, as with `STARTS_WITH`. ## Engine wiring Spark 3.5 / 4.0 / 4.1 `SparkV2Filters`. Other engine bindings are unaffected until their converters add the mapping. ## Testing Mirrors the existing `STARTS_WITH` coverage: `TestEvaluator`, `TestExpressionHelpers` (rewriteNot/negate), `TestExpressionBinding` (incl. non-string rejection), `TestInclusiveMetricsEvaluator`, `TestStrictMetricsEvaluator`, `TestInclusiveManifestEvaluator`, `TestExpressionUtil` (sanitize), `TestExpressionSerialization`, `TestExpressionParser` round-trip, and `TestSparkV2Filters` for each Spark version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
