I would like to discuss a problem I am facing upgrading Beam 2.24.0 -> 2.33.0.
Running Beam batch jobs on SparkRunner with Spark 2.4.4 stopped showing me job details on Spark History Server. Problem is that there are 2 event logging. listener running and they step on each other. More details in [1]. One is run by Spark itself, the other is started by Beam, which was added by MR [2]. My first question is towards understanding why there is Spark's even logging listener started manually within Beam next to the one started by Spark Context internally? [1] https://issues.apache.org/jira/browse/BEAM-13981 [2] https://github.com/apache/beam/pull/14409
