1996fanrui commented on code in PR #827:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/827#discussion_r1596344772
##########
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java:
##########
@@ -247,17 +249,36 @@ protected JobTopology getJobTopology(JobDetailsInfo
jobDetailsInfo) {
json, slotSharingGroupIdMap, maxParallelismMap, metrics,
finished);
}
- private void updateKafkaSourceMaxParallelisms(Context ctx, JobID jobId,
JobTopology topology)
- throws Exception {
+ private void updateKafkaPulsarSourceMaxParallelisms(
+ Context ctx, JobID jobId, JobTopology topology) throws Exception {
try (var restClient = ctx.getRestClusterClient()) {
- var partitionRegex =
Pattern.compile("^.*\\.partition\\.\\d+\\.currentOffset$");
+ Pattern partitionRegex =
+ Pattern.compile(
+
"^.*\\.KafkaSourceReader\\.topic\\.(?<kafkaTopic>.+)\\.partition\\.(?<kafkaId>\\d+)\\.currentOffset$"
+ +
"|^.*\\.PulsarConsumer\\.(?<pulsarTopic>.+)-partition-(?<pulsarId>\\d+)\\..*\\.numMsgsReceived$");
for (var vertexInfo : topology.getVertexInfos().values()) {
if (vertexInfo.getInputs().isEmpty()) {
var sourceVertex = vertexInfo.getId();
var numPartitions =
queryAggregatedMetricNames(restClient, jobId,
sourceVertex).stream()
- .filter(partitionRegex.asMatchPredicate())
- .count();
+ .map(
Review Comment:
Sorry, I got it from your description: `Since Pulsar Source metrics are
related to a single Consumer identifier, a single partition may have multiple
consumer identifiers, so some deduplication needs to be done here, grouping by
topic and partition ID.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
