lgbo-ustc opened a new pull request, #12417: URL: https://github.com/apache/gluten/pull/12417
## What changes were proposed in this pull request? Support checkpoint/restore for Gluten Flink native Kafka source offsets: 1. **Persist native source checkpoint records**: `GlutenSourceFunction` now snapshots native source state with the real Flink checkpoint id and stores the returned records in Flink operator `ListState<String>`. 2. **Restore source offsets from operator state**: restored source checkpoint records are loaded from Flink operator state and passed back to native execution during source initialization. 3. **Kafka source restore coverage**: add a Kafka + MiniCluster IT that verifies checkpoint records are written, restored after failure, and can be manually bootstrapped to start consuming from a specified offset. 4. **Kafka source helper**: add a test helper for reading and writing Gluten Kafka source checkpoint state through Flink State Processor API. 5. **Kafka source config fixes**: align native Kafka startup offset handling with Flink startup mode and pass `task_parallelism` expected by the native Kafka connector. Depends on: - bigo-sg/velox4j `feature/flink-source-offset-checkpoint-state` (`7bea365`) ## How was this patch tested? ```bash mvn spotless:apply -pl ut mvn test -pl ut -am -Dtest=KafkaSourceCheckpointIT -DfailIfNoTests=false ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI gpt-5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
