HeartSaVioR commented on pull request #31471: URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959
Actually I have been thinking about this - while I think this helps to track down the elapsed time on committing job, there's still another problem end users confuse the huge delay of commit job with being stuck unless they took the stack trace. Unfortunately Hadoop layer leaves log messages with DEBUG level, hence no log message may be written during commit phase. (I've added latency information on DEBUG log message in Hadoop side via MAPREDUCE-7317, but will be available in 3.3.1+ and the log level is still DEBUG.) Probably we'd need to print out some informative log message during commit phase periodically - not sure where is the right place to fix, Hadoop vs Spark. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
