Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/23260
@srowen
For now executor log url is **static** in Spark, which forces Node Manager
to be alive even after application is finished, in order to provide executor
log in SHS.
This situation can be happen when decommission happens a bit frequently, or
when end users want a kind of elasticity against YARN cluster (not only
decommissioning nodes, but also elasticity on YARN cluster itself - YARN has
cluster id for RM which classifies the cluster which can be leveraged when
dealing with multiple YARN clusters.)
There's also similar change applied on Hadoop side.
https://github.com/apache/hadoop/commit/5fe1dbf1959976d0dc5a8e614dd74836cfbee04c
We are experimenting central log service which resolves above situation. At
least the log url for centralized log service can't be same URL as NM webapp,
we have to get flexibility of executor log URL.
Hope it explains the rationalization well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
