danzhewuju opened a new issue, #11066:
URL: https://github.com/apache/paimon/issues/11066

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   paimon 2.2-SNAPSHOT
   
   ### Compute Engine
   
   Spark 3.5.9
   
   ### Minimal reproduce step
   
   An application LEFT JOINs a request list with a Paimon table to obtain 
candidate row IDs, then uses those IDs in a second Spark SQL query. An 
unmatched request produces a NULL row ID. Including that NULL in the generated 
IN filter causes the second query to fail.
   
   **1. Create a Data Evolution table with two orders.**
   ```sql
   CREATE TABLE paimon.default.orders (
       order_id BIGINT,
       payload STRING
   ) USING paimon
   TBLPROPERTIES (
       'bucket' = '-1',
       'row-tracking.enabled' = 'true',
       'data-evolution.enabled' = 'true'
   );
   INSERT INTO paimon.default.orders VALUES
       (100L, 'order-100'), (200L, 'order-200');
   ```
   **2. Resolve row IDs for the requested orders, including an order that does 
not exist.**
   ```sql
   WITH requests AS (
       SELECT * FROM VALUES
           (0, 100L), (1, 999L), (2, 200L)
           AS r(pos, order_id)
   )
   SELECT p._ROW_ID
   FROM requests r
   LEFT JOIN paimon.default.orders p ON r.order_id = p.order_id
   ORDER BY r.pos;
   ```
   This query succeeds and returns `[0, NULL, 1]` in the local run. Paimon 
assigned the non-null IDs 0 and 1 to the stored orders; the LEFT JOIN 
introduces NULL for the missing order 999. Use the actual returned IDs if they 
differ.
   **3. Use the candidate IDs in the next Spark SQL query.**
   ```sql
   SELECT order_id
   FROM paimon.default.orders
   WHERE _ROW_ID IN (0L, CAST(NULL AS BIGINT), 1L);
   ```
   Expected: orders 100 and 200 are returned. Actual: planning fails with a 
`SparkException` caused by a `NullPointerException` in `RowIdPredicateVisitor`. 
Removing the NULL member makes the query succeed.
   
   ### What doesn't meet your expectations?
   
   **Expected:** a NULL IN member contributes no matching rows, while the 
non-null IDs still match. Both the SQL and DataFrame queries should return 
orders 100 and 200. This agrees with the existing Paimon `In` row evaluator, 
which skips null literals.
   **Actual:** Spark wraps the Paimon exception in a planning error:
   ```text
   org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase 
planning failed with an internal error.
   Caused by: java.lang.NullPointerException
       at org.apache.paimon.utils.Range.toRanges(Range.java:205)
       at 
org.apache.paimon.predicate.RowIdPredicateVisitor.visit(RowIdPredicateVisitor.java:61)
       ...
       at 
org.apache.paimon.globalindex.DataEvolutionBatchScan.withFilter(DataEvolutionBatchScan.java:106)
       at 
org.apache.paimon.table.source.ReadBuilderImpl.configureScan(ReadBuilderImpl.java:222)
       at 
org.apache.paimon.table.source.ReadBuilderImpl.newScan(ReadBuilderImpl.java:203)
       at 
org.apache.paimon.spark.PaimonBaseScan.getInputSplits(PaimonBaseScan.scala:53)
       ...
       at org.apache.paimon.spark.read.BaseScan.toBatch(BaseScan.scala:185)
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to