Hey Morgan, I would query the Monitoring REST API: https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html
For example: GET http://localhost:8082/jobs/9a6748889bf24987495eead247aeb1ff Returns: 1. {jid: "9a6748889bf24987495eead247aeb1ff", name: "CarTopSpeedWindowingExample", isStoppable: false,…} 1. jid: "9a6748889bf24987495eead247aeb1ff" 2. name: "CarTopSpeedWindowingExample" 3. isStoppable: false 4. state: "RUNNING" 5. start-time: 1582192403413 6. end-time: -1 7. duration: 18533 8. now: 1582192421946 9. timestamps: {FINISHED: 0, FAILING: 0, CANCELED: 0, SUSPENDED: 0, RUNNING: 1582192403550, RECONCILING: 0, FAILED: 0,…} 10. vertices: [{id: "cbc357ccb763df2852fee8c4fc7d55f2", name: "Source: Custom Source -> Timestamps/Watermarks",…},…] 1. 0: {id: "cbc357ccb763df2852fee8c4fc7d55f2", name: "Source: Custom Source -> Timestamps/Watermarks",…} 1. id: "cbc357ccb763df2852fee8c4fc7d55f2" 2. name: "Source: Custom Source -> Timestamps/Watermarks" 3. parallelism: 1 4. status: "RUNNING" 5. start-time: 1582192403754 6. end-time: -1 7. duration: 18192 8. tasks: {CREATED: 0, CANCELED: 0, RECONCILING: 0, FAILED: 0, CANCELING: 0, DEPLOYING: 0, RUNNING: 1,…} 9. metrics: {read-bytes: 0, read-bytes-complete: true, write-bytes: 0, write-bytes-complete: true, read-records: 0,…} 2. 1: {id: "90bea66de1c231edf33913ecd54406c1",…} 1. id: "90bea66de1c231edf33913ecd54406c1" 2. name: "Window(GlobalWindows(), DeltaTrigger, TimeEvictor, ComparableAggregator, PassThroughWindowFunction) -> Sink: Print to Std. Out " 3. parallelism: 1 4. status: "RUNNING" 5. start-time: 1582192403759 6. end-time: -1 7. duration: 18187 8. tasks: {CREATED: 0, CANCELED: 0, RECONCILING: 0, FAILED: 0, CANCELING: 0, DEPLOYING: 0, RUNNING: 1,…} 9. metrics: {read-bytes: 4669, read-bytes-complete: true, write-bytes: 0, write-bytes-complete: true,…} 11. status-counts: {CREATED: 0, CANCELED: 0, RECONCILING: 0, FAILED: 0, CANCELING: 0, DEPLOYING: 0, RUNNING: 2,…} 12. plan: {jid: "9a6748889bf24987495eead247aeb1ff", name: "CarTopSpeedWindowingExample",…} On Tue, Feb 18, 2020 at 5:01 PM Morgan Geldenhuys < [email protected]> wrote: > Hi All, > > I have setup monitoring for Flink (1.9.2) via Prometheus and am interested > in viewing the end-to-end latency at the sink operators for the 95 > percentile. I have enabled latency markers at the operator level and can > see the results, one of the entries looks as follows: > > > flink_taskmanager_job_latency_source_id_operator_id_operator_subtask_index_latency{app="flink",component="taskmanager",host="flink_taskmanager_6bdc8fc49_kr4bs",instance=" > 10.244.18.2:9999 > ",job="kubernetes-pods",job_id="96d32d8e380dc267bd69403fd7e20adf",job_name="Traffic",kubernetes_namespace="default",kubernetes_pod_name="flink-taskmanager-6bdc8fc49-kr4bs",operator_id="2e32dc82f03b1df764824a4773219c97",operator_subtask_index="7",pod_template_hash="6bdc8fc49",quantile="0.95",source_id="cbc357ccb763df2852fee8c4fc7d55f2",tm_id="7fb02c0ed734ed1815fa51373457434f"} > > That is great, however... I am unable to determine which of the operators > is the sink operator I'm looking for based solely on the operator_id. Is > there a way of determining this? > > Regards, > M. >
