[GitHub] [spark] mridulm commented on pull request #38711: [SPARK-41192][CORE] Remove unscheduled speculative tasks when task finished to obtain better dynamic
mridulm commented on PR #38711: URL: https://github.com/apache/spark/pull/38711#issuecomment-1360815556 Merged to master. Thanks for working on this @toujours33 ! Thanks for the reviews @LuciferYang, @Ngone51, @dongjoon-hyun :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #38711: [SPARK-41192][Core] Remove unscheduled speculative tasks when task finished to obtain better dynamic
mridulm commented on PR #38711: URL: https://github.com/apache/spark/pull/38711#issuecomment-1322379454 We need `stageAttemptToSpeculativeTaskIndices` for `removeStageFromResourceProfileIfUnused` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #38711: [SPARK-41192][Core] Remove unscheduled speculative tasks when task finished to obtain better dynamic
mridulm commented on PR #38711: URL: https://github.com/apache/spark/pull/38711#issuecomment-1321576092 Thanks for the PR to fix this bug ! I will need to think more about this issue, but I am leaning towards a variant of the solution proposed. Namely: * `stageAttemptToNumSpeculativeTasks` -> `stageAttemptToUnsubmittedSpeculativeTasks` * These are task indices which have been marked speculatable, which have not yet been launched. * When a speculative task starts, remove it from `stageAttemptToUnsubmittedSpeculativeTasks` * When a non-speculative task finishes, remove it from `stageAttemptToUnsubmittedSpeculativeTasks` Thoughts @Ngone51, @toujours33, @LuciferYang ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
