Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3486#issuecomment-66363953
Hey @ksakellis I think we should explore a different approach that doesn't
involve adding new listener events. Have you considered something like the
following:
The AM is the one that launches these executors, and it has global
knowledge of which executor maps to which container IDs and thus log URLs. The
next step then is to pass this knowledge to the driver, which contains the UI.
In cluster mode, this is easy because the AM already has a reference to
`sc`, which is launched within the same JVM. In client mode, there is an
existing communication path through the `AMActor` in `ApplicationMaster`.
Instead of passing these URLs through env vars and listeners we can just have
the `AMActor` send a message to the driver every time a new executor is
launched. This bypasses passing the log URLs to the executors themselves, which
don't consume them except for passing them along to the driver.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
