danzhewuju opened a new issue, #11066: URL: https://github.com/apache/paimon/issues/11066
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version paimon 2.2-SNAPSHOT ### Compute Engine Spark 3.5.9 ### Minimal reproduce step An application LEFT JOINs a request list with a Paimon table to obtain candidate row IDs, then uses those IDs in a second Spark SQL query. An unmatched request produces a NULL row ID. Including that NULL in the generated IN filter causes the second query to fail. **1. Create a Data Evolution table with two orders.** ```sql CREATE TABLE paimon.default.orders ( order_id BIGINT, payload STRING ) USING paimon TBLPROPERTIES ( 'bucket' = '-1', 'row-tracking.enabled' = 'true', 'data-evolution.enabled' = 'true' ); INSERT INTO paimon.default.orders VALUES (100L, 'order-100'), (200L, 'order-200'); ``` **2. Resolve row IDs for the requested orders, including an order that does not exist.** ```sql WITH requests AS ( SELECT * FROM VALUES (0, 100L), (1, 999L), (2, 200L) AS r(pos, order_id) ) SELECT p._ROW_ID FROM requests r LEFT JOIN paimon.default.orders p ON r.order_id = p.order_id ORDER BY r.pos; ``` This query succeeds and returns `[0, NULL, 1]` in the local run. Paimon assigned the non-null IDs 0 and 1 to the stored orders; the LEFT JOIN introduces NULL for the missing order 999. Use the actual returned IDs if they differ. **3. Use the candidate IDs in the next Spark SQL query.** ```sql SELECT order_id FROM paimon.default.orders WHERE _ROW_ID IN (0L, CAST(NULL AS BIGINT), 1L); ``` Expected: orders 100 and 200 are returned. Actual: planning fails with a `SparkException` caused by a `NullPointerException` in `RowIdPredicateVisitor`. Removing the NULL member makes the query succeed. ### What doesn't meet your expectations? **Expected:** a NULL IN member contributes no matching rows, while the non-null IDs still match. Both the SQL and DataFrame queries should return orders 100 and 200. This agrees with the existing Paimon `In` row evaluator, which skips null literals. **Actual:** Spark wraps the Paimon exception in a planning error: ```text org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. Caused by: java.lang.NullPointerException at org.apache.paimon.utils.Range.toRanges(Range.java:205) at org.apache.paimon.predicate.RowIdPredicateVisitor.visit(RowIdPredicateVisitor.java:61) ... at org.apache.paimon.globalindex.DataEvolutionBatchScan.withFilter(DataEvolutionBatchScan.java:106) at org.apache.paimon.table.source.ReadBuilderImpl.configureScan(ReadBuilderImpl.java:222) at org.apache.paimon.table.source.ReadBuilderImpl.newScan(ReadBuilderImpl.java:203) at org.apache.paimon.spark.PaimonBaseScan.getInputSplits(PaimonBaseScan.scala:53) ... at org.apache.paimon.spark.read.BaseScan.toBatch(BaseScan.scala:185) ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
