grishaw commented on PR #46696: URL: https://github.com/apache/spark/pull/46696#issuecomment-2461577555
Hi, I'm hoping someone can assist with a question I have. We are currently using EMR-6.13.0 with Spark 3.4.1 and are experiencing issues related to the "Authorized committer error". I know that there are fixes available in Spark versions 3.4.4 and 3.5.2, but neither of these versions is currently available on the EMR platform. As a workaround, we are considering disabling the OutputCommitCoordinator by setting "spark.hadoop.outputCommitCoordination.enabled" to "false". My question is, if we are willing to accept occasional duplicates, is it safe to disable the OutputCommitCoordinator? Our main concern is the possibility of data loss — could that occur if we disable this feature? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
