szehon-ho commented on PR #17956: URL: https://github.com/apache/iceberg/pull/17956#issuecomment-5723911727
Fixed UUID partition-key conversion in b6a0c9771e. The adapter now reuses `SparkUtil.internalToSpark`, so UUID values are supplied as Spark strings. The new regression test checks both matching rows and the number of input partitions under local and distributed planning. Both cases failed before the fix and pass now. I also checked the failure-handling issue in apache/spark#58851. This PR returns every opaque predicate from `pushPredicates`, so Spark retains the original post-scan filter. On Spark 4.2.0, treating an evaluation failure as a match can only reduce pruning; rows still pass through the original filter. With the Spark fix, evaluation failures propagate out of task planning. `SparkTable.canDeleteWhere` also rejects opaque predicates, so these deletes fall back to row-level execution. The metadata-only DELETE data-loss path from the Spark issue is therefore not enabled here. Validation on Spark 4.2.0 / Java 17: - `TestFilterPushDown`: 42 tests passed, including both UUID regression cases. - `TestFilteredScan`: 76 passed, 9 skipped. - Existing `StructInternalRow` suites: 9 tests passed. - Spark 4.2 formatting checks passed. The Spark 4.2.1 behavior was checked in code; these tests ran against 4.2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
