Alpha162 commented on PR #13927: URL: https://github.com/apache/cloudstack/pull/13927#issuecomment-5371368226
Thanks for picking this up, and for the detail in the description. Two corrections on the figures which I updated on #13906 after restoring a pre-rebuild backup: it was 1,676 consecutive successful jobs pinned to start_date = 2026-06-04 16:50:51, and exec_time peaked at 3,139,799 ms, so 52 minutes. The 54.4M rows and the #13112 cross-reference both hold. @DaanHoogland's question about a longer window is worth pinning down, because the window length sets a cost. With a permanently unprocessable event, Math.max picks startDateMillis - MAX on every run, so each aggregation period gets re-covered once per run until it drops out of the window: roughly 24x at a day, around 168x at a week. Bounded either way, which is the win here, but the multiplier scales with the bound. The other thing left over is that a stuck event stays stuck and stays quiet. The job still records success = 1, and usage.sanity.check.interval defaults to NULL. On our cluster the fix was forcing the six affected rows to processed = 1, which took the hourly job from 52 minutes to 2.1 seconds. None of that argues against this change. Happy to raise the quarantine and surfacing side as a separate issue if you'd rather keep this one focused. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
