zhouyejoe commented on pull request #34018: URL: https://github.com/apache/spark/pull/34018#issuecomment-922142969
I am also thinking about the same. > Discussed with @Ngone51. > @zhouyejoe can we make the following change please: > > * Move toInt from SparkContext into setAttemptId (which will take it as a `String` param). > * try/catch the `Integer.parseInt` and log warning in case it is not able to parse attempt id to string. > > * If successful, set attempt id to that value - else leave it unchanged (defaulting to -1). > * Let us keep the attempt id in BlockStoreClient itself. > > Thoughts ? Update: 1. Changed the AppAttemptId type in BlockStoreClient from Int to String. Add integer comparableAppAttemptId in ExternalBlockStoreClient, and override the getAppAttemptId and setAppAttemptId there. 2. Try catch the String to Integer parse in setComparableAppAttemptId. 3. The default value of comparableAppAttemptId is -1. 4. Only setAppAttemptId in Driver or Executor when Push based shuffle is enabled. 5. Minor change in PushBasedExternalClusterManager to return 1 for the TaskScheduler.applicationAttemptId() -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
