Hello Jozef this change was not introduced in the PR you referenced, that PR was just a refactor. The conflicting change was added in [1] via [2] starting on Beam 2.29.0.
It is not clear for me why this was done but maybe Kyle Weaver or someone else have a better context. Let's continue the discussion on the issue. For me the right solution is to remove the extra event logging listener but I might be missing something, in all cases it does not seem the runner is the best place to deal with this logic, this seems error-prone. [1] https://github.com/apache/beam/pull/13743/ [2] https://github.com/apache/beam/commit/291ced166af On Wed, Feb 23, 2022 at 3:41 PM Jozef Vilcek <jozo.vil...@gmail.com> wrote: > > I would like to discuss a problem I am facing upgrading Beam 2.24.0 -> 2.33.0. > > Running Beam batch jobs on SparkRunner with Spark 2.4.4 stopped showing me > job details on Spark History Server. Problem is that there are 2 event > logging. listener running and they step on each other. More details in [1]. > One is run by Spark itself, the other is started by Beam, which was added by > MR [2]. > > My first question is towards understanding why there is Spark's even logging > listener started manually within Beam next to the one started by Spark > Context internally? > > > [1] https://issues.apache.org/jira/browse/BEAM-13981 > [2] https://github.com/apache/beam/pull/14409 > >
