HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-453805982 Sorry to revisit this lately: I had to tackle some other tasks as well, so spent time to do simpler thing first. @vanzin I guess I'm confusing based on your previous comment: > The parameters don't need to be pre-defined, each RM can have its own set, and it's up to the admin to set things up so that the URLs in the SHS make sense for their deployment. especially `The parameters don't need to be pre-defined`. In this patch, I defined some parameters which I think they can make a reference of specific container being unique. I interpreted `no pre-defined` as flexibility, so tried to find a way to let end users (or admin) defines their own set of parameters, hence thought about interface and plug-in, like having interface which inputs will be YarnConfiguration, Container, etc (for YARN, others should have different interface). If we are happy with pre-defining parameters by ourselves and let end users just be flexible with constructing log URL, things would be fairly simpler. And I guess that looks like what you really meant, by part of your comment below: > All that's missing is writing those to the event log somehow, and using them on the SHS side, instead of doing it all on the application side, which has the drawbacks that have already been discussed. Could you please confirm I understand correctly?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
