LuciferYang opened a new issue, #10939:
URL: https://github.com/apache/paimon/issues/10939

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (30d91278af)
   
   ### Compute Engine
   
   Flink
   
   ### Minimal reproduce step
   
   A REST catalog table with `'query-auth.enabled' = 'true'` and a row filter 
returned by the server, read in streaming mode with snapshot-aligned 
checkpoints:
   
   ```sql
   SELECT * FROM t /*+ OPTIONS('source.checkpoint-align.enabled' = 'true') */;
   ```
   
   while another job keeps committing to `t`.
   
   ### What doesn't meet your expectations?
   
   The streaming job fails. If `t` has no snapshot when the job starts:
   
   ```
   java.lang.NullPointerException
        at java.util.Objects.requireNonNull(Objects.java:203)
        at 
org.apache.paimon.flink.source.align.AlignedContinuousFileSplitEnumerator.processDiscoveredSplits(AlignedContinuousFileSplitEnumerator.java:249)
   ```
   
   If `t` already has data, it fails when the enumerator assigns a placeholder 
split:
   
   ```
   java.lang.NullPointerException
        at org.apache.paimon.table.source.DataSplit.toString(DataSplit.java:379)
        at 
org.apache.paimon.flink.source.FileStoreSourceSplit.toString(FileStoreSourceSplit.java:85)
        ...
        at 
org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext.assignSplits(SourceCoordinatorContext.java:304)
        at 
org.apache.paimon.flink.source.ContinuousFileSplitEnumerator.assignSplits(ContinuousFileSplitEnumerator.java:333)
   ```
   
   The same read without a row filter works.
   
   A stream scan returns `SnapshotNotExistPlan.INSTANCE` when there is no new 
snapshot, and its consumers check for that instance: 
`AlignedContinuousFileSplitEnumerator` skips it, 
`ContinuousFileSplitEnumerator` stops triggering scans, Spark's `StreamHelper` 
and `CDCSourceEnumerator` check it too. When the table has query auth rules, 
`TableQueryAuthResult.convertPlan` wraps every plan into a new `DataFilePlan`, 
including `SnapshotNotExistPlan`, so none of them recognize it. The aligned 
enumerator then queues every empty poll as a pending plan, with a null next 
snapshot id while the table is empty, and otherwise as a placeholder split.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to