lnbest0707 opened a new pull request, #277:
URL: https://github.com/apache/flink-connector-kafka/pull/277
## What is the purpose of the change
`DynamicKafkaSource` currently exposes cluster- and split-scoped metrics
whose metric groups can
disappear when dynamic metadata removes clusters or partitions. Consumers
such as the Flink
Kubernetes Operator autoscaler need a stable per-reader signal for the
current active split
assignment.
This change adds a lifetime `DynamicKafkaSource.activeSplitCount` gauge for
each dynamic source
reader. It reports `0` before assignment and after all active splits are
removed. The count is
derived only from splits assigned to active underlying `KafkaSourceReader`s,
so pending restored
state and retained removed-cluster checkpoint offsets are excluded. The
cached atomic value keeps
metric reporter threads from traversing mutable reader state.
## Brief change log
- register `DynamicKafkaSource.activeSplitCount` at the stable dynamic
reader metric root
- refresh the count after split assignment, metadata rebuild/removal,
polling, and reader close
- add coverage for empty readers, assignment, partial/all metadata removal,
retained-state
exclusion, and reactivation
- keep the existing cluster-metric lifecycle test scoped to cluster metrics
- document the gauge in the English and Chinese Dynamic Kafka source docs
## Verifying this change
```bash
DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock \
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock \
TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 \
mvn -pl flink-connector-kafka \
-Dtest=DynamicKafkaSourceReaderTest \
-Drat.skip=true clean test
DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock \
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock \
TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 \
mvn -pl flink-connector-kafka \
-Dtest='DynamicKafkaSourceITTest$DynamicKafkaSourceSpecificTests#testMetricsLifecycleManagement'
\
-Drat.skip=true test
DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock \
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock \
TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 \
mvn -pl flink-connector-kafka -Drat.skip=true clean test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
