squito commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-455616927 >> The one thing I am wondering about is if we should change it so that when the SHS is looking at an inprogress file, it does not do any replacement. > I'm sorry but could you elaborate? I'm honestly not getting familiar with UI/SHS codebase, so I can be missing here. I'm thinking of running with yarn with having the logs be available in their regular location while the app is running, but only after it completes and yarn logs have been aggregated, redirecting the user to the JHS. In that case, you want the live UI to show the original URLs, and also even the SHS to show the original URLs, *until* the application completes. Only after the application completes, then you want the SHS to do the replacement you have, to point to the JHS (because the JHS won't have any logs until after the app finishes). This may be different than your use case, where you have some other log storage mechanism which can display the logs even while the application is still running, and you want the SHS to *always* replace the URLs, as your system makes the logs visible even while the app is still running.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
