[
https://issues.apache.org/jira/browse/FLINK-39617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106384#comment-18106384
]
Zhe Wang commented on FLINK-39617:
----------------------------------
Thanks for opening this for API-shape discussion.
The purely additive direction sounds reasonable to me. I checked the existing
aggregated subtask metrics handler, and keeping the existing single-vertex
endpoint unchanged while adding separate batch endpoints for names and values
looks like a good fit.
One detail worth clarifying is the regex semantics for the metric-name
endpoint. Should the regex be treated as a full Java regex match, where users
would write patterns like `.*busyTime.*`, or as a substring-style match? I
slightly prefer making this explicit in the API contract.
I also think keeping the names and values endpoints separate is cleaner than
combining them, since the values request naturally needs per-vertex metric
lists and aggregation options, while the names request only needs vertex IDs
plus optional filters.
I have done some local source-level exploration to validate that this can be
implemented additively by reusing the existing aggregation logic, but I will
wait for API-shape consensus before opening a PR. If this direction looks good
and no one else is already working on the implementation, I would be happy to
help submit it.
Thanks,
Zhe Wang
> Add batch REST endpoints for aggregated subtask metrics across multiple job
> vertices
> --------------------------------------------------------------------------------------
>
> Key: FLINK-39617
> URL: https://issues.apache.org/jira/browse/FLINK-39617
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / REST
> Reporter: Herbert Wang
> Priority: Major
> Labels: Metrics, metrics, rest_api
>
> The JobManager REST API currently exposes aggregated subtask metrics per job
> vertex via:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients that need the same metric set for many vertices, such as autoscalers
> or monitoring integrations, must issue one request per vertex for metric-name
> discovery and another request per vertex for metric values. For jobs with
> many vertices this creates avoidable REST fan-out, repeated MetricFetcher
> updates, and large repeated payloads.
> h2. Proposal
> Add two batch JobManager REST endpoints for aggregated subtask metrics across
> multiple vertices:
> {code}
> POST /jobs/:jobid/vertices/subtasks/metrics/names
> POST /jobs/:jobid/vertices/subtasks/metrics/values
> {code}
> The existing single-vertex endpoint should remain unchanged for compatibility.
> The endpoints are intentionally split rather than using one POST endpoint
> with mode-switching behavior, so OpenAPI schemas, code generation, and
> capability detection remain straightforward.
> h3. Name discovery endpoint
> Request:
> {code:json}
> {
> "vertexIds": ["<jobVertexId>", "<jobVertexId>"],
> "regex": [".*busyTime.*", ".*numRecords.*"]
> }
> {code}
> Response:
> {code:json}
> [
> {
> "vertexId": "<jobVertexId>",
> "metrics": [{ "id": "busyTimeMsPerSecond" }]
> }
> ]
> {code}
> h3. Value aggregation endpoint
> Request:
> {code:json}
> {
> "vertices": [
> { "vertexId": "<jobVertexId>", "metrics": ["busyTimeMsPerSecond"] },
> { "vertexId": "<jobVertexId>", "metrics": ["numRecordsInPerSecond"] }
> ],
> "agg": ["min", "max", "avg"]
> }
> {code}
> Response:
> {code:json}
> [
> {
> "vertexId": "<jobVertexId>",
> "metrics": [{ "id": "busyTimeMsPerSecond", "min": 0.0, "max": 1.0, "avg":
> 0.5 }]
> }
> ]
> {code}
> h2. Compatibility
> This is additive. The existing endpoint remains unchanged:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients can feature-detect the new endpoints and fall back to the existing
> per-vertex endpoint when unavailable, or we can cherry-pick to earlier 2.x
> versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
