JingsongLi opened a new pull request, #9306: URL: https://github.com/apache/paimon/pull/9306
## What changed - add a Catalyst optimizer bridge for literal `array_contains` and `arrays_overlap` predicates on Paimon tables - translate the bridged Spark V2 predicates into Paimon `ARRAY_CONTAINS` and `ARRAYS_OVERLAP` predicates - convert Spark array literals into Paimon element literals - verify that eligible predicates produce `IndexedSplit` instances backed by the multivalue index - keep non-literal predicates as Spark residual filters and preserve null semantics ## Why Spark's V2 expression builder does not translate Catalyst `ArrayContains` or `ArraysOverlap` expressions. As a result, Paimon only received the generated `IS_NOT_NULL` predicate, so an existing multivalue index could not be selected for array membership queries. This change exposes literal array predicates through Spark's generic V2 predicate path while retaining the original Catalyst evaluation for residual filtering. ## Validation - Spark 3 `PrimaryKeySortedIndexTest`: 3 tests passed - Spark 4.1 `SparkV2FilterConverterTest`: 30 tests passed - Spark 4.1: the new multivalue array predicate test passed; the same class run also encountered a local Avro binary mismatch in an unrelated existing postpone-bucket test - Spotless checks passed for the Spark common and UT modules - `git diff --check` passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
