zhanglistar opened a new pull request, #12318:
URL: https://github.com/apache/gluten/pull/12318
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
This PR wires native source checkpoint state through Gluten Flink and fixes
task-level output metrics for Gluten streaming operators. Depends on
https://github.com/bigo-sg/velox/pull/47 and
https://github.com/bigo-sg/velox4j/pull/37.
Add Gluten source checkpoint state persistence through Flink operator
ListState.
Pass real Flink checkpoint IDs into native snapshot/complete/abort paths.
Restore native checkpoint records during source initialization.
Add native source metrics fallback for unique TableScan stats when Flink
operator ID does not match the Velox plan node ID.
Fix Gluten one-input and two-input operators to update task-level
numRecordsOut based on the actual number of emitted rows.
Change VectorOutputBridge.collect() to return emitted record count so task
metrics reflect row output accurately.
Why
Kafka source checkpointing needs native progress snapshots to be persisted
by Flink and restored on failover. Also, Kafka -> Gluten calc -> blackhole jobs
previously showed vertex-level write-records = 0 even when Gluten operator
metrics were non-zero, because task-level output counters were not updated by
the Gluten output path.
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
## How was this patch tested?
Validation
```
mvn -pl runtime,ut -am -Dtest=GlutenStreamFilterTest,SourceTaskMetricsTest
-DfailIfNoTests=false test
mvn -pl runtime,loader -am -DskipTests -DfailIfNoTests=false package
```
Ran local Flink 1.19.2 Kafka smoke:Kafka source produced 5 records
Gluten calc emitted 3 records after filter
Flink REST vertex metrics showed write-records = 3
Job remained RUNNING
Checkpoints completed successfully.
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
co work with codex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
