SEZ9 commented on issue #12123: URL: https://github.com/apache/seatunnel/issues/12123#issuecomment-6407156875
Thanks @awsomesud347 for digging into this and for correcting the original framing. The key finding — that `/running-jobs` and `/finished-jobs` go through `JobInfoService` directly and never hit `JobHistoryService.getJobStatusData()` — means the server-side paging/filtering I originally suggested for the listing endpoints would not have addressed the real cost. Good that #12130 covers the part it does, and that the retention/TTL side stays with #10767. Also useful that you ruled out the "avoid building `JobStatusData` per job" approach with numbers (198.66 vs 198.71 MB/op, ~0.19% latency delta). That saves anyone else from retrying it, and confirms the IMap fetch itself is what needs to change on the `/metrics` and `/openmetrics` scrape path. @goutamadwant thanks for the pointer to #12982 for the member-side aggregation. I'll review it there. A few things I'd like to see confirmed in that PR, since this is exactly the piece @awsomesud347 flagged as untested: 1. Measurements on a multi-member cluster, not just single-member. As noted above, a single-member setup has nothing crossing the network, so it cannot show whether aggregating counts member-side actually reduces the per-scrape cost compared to the ~200 MB/op baseline for 10k retained jobs. 2. Confirmation that the aggregated status counts exposed via `/metrics` and `/openmetrics` are identical to the current output for the same job set (no changes to which states are counted or how). 3. A note on behavior when a member is unavailable or joins mid-scrape, so we understand whether counts can be partial and how that is surfaced. If the PR description already covers these, a short pointer is enough. Otherwise, let's get them added before we move forward. <!-- streview-comment:2041 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
