sajjad-moradi commented on PR #13968: URL: https://github.com/apache/pinot/pull/13968#issuecomment-2347265151
You are right. This has some ZK read overhead, and it should be solved on the push side. FYI, the problem on the push side is that although Spark driver provides a way to synchronously cancel all spun off jobs, in practice, the actual job cancellation is not 100% synchronous, so we might end up having inflight push jobs. I decided not to call revert for failure/cancel cases in the same job. The next job will revert the in_progress entry, or retention manager will clean it up. Closing the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
