I agree that, ideally, users should delegate control of the logging backend to the runner or SDK harness to ensure logs are correctly captured. Dataflow, as you know, has a mechanism for collecting logs written by the JUL backend (via the default slf4j-jdk14 bridge) and sending them to Cloud Logging. This approach may not work natively with other logging backends. For example, when slf4j-simple is used, logs will be written to stdout or stderr, and eventually appear in a different Cloud Logging category.
However, we have received customer support tickets (internal bug IDs: 307965737, 319291955, and 381398168) requesting support for alternative backends like Log4j or Logback. These customers often have transitive Log4j or Logback dependencies and user-customized logging configurations within their pipelines that are difficult to get rid of. Rather than failing their pipeline runs (e.g., a crash loop can happen when users have both the log4j-slf4j-impl bridge and the Log4j backend, due to our prepending log4j-to-slf4j to the classpath) or forcing the default JUL backend (due to the previous higher priority of slf4j-jdk14 on the classpath), we propose allowing them to use their own dependencies at their own discretion and risk. Honestly, if we want to fully support other logging backends in Dataflow and Cloud Logging, there will be some adjustments to the Dataflow codebase. On Wed, Feb 5, 2025 at 10:04 AM Kenneth Knowles <k...@apache.org> wrote: > Thanks for the very nice document. Logging has been a source of > complexity since the beginning of Beam, and upgrading to SLF4J2 is getting > urgent. > > I do have big picture questions/concerns: > > - For non-portable runners/pipelines, shouldn't the runner totally > control logging backends so that it can capture logs correctly? > > - For portable runners, shouldn't the SDK harness totally control logging > backends so it can ship logs out of the container correctly? > > Can you explain the intended use of having the user override the backend? > (I totally understand that since we don't control dependencies we need all > the frontend bridges) > > Kenn > > On Tue, Feb 4, 2025 at 3:15 PM Shunping Huang <mark.sphu...@gmail.com> > wrote: > >> Hi everyone, >> >> I put together a short doc to summarize the existing logging >> infrastructure(dependencies) in Beam Java and outline a plan to improve it. >> Basically, we are on the path towards slf4j 2.x. >> >> >> https://docs.google.com/document/d/1IkbiM4m8D-aB3NYI1aErFZHt6M7BQ-8eCULh284Davs/edit?usp=sharing >> >> If you are interested in this topic, please take a look and share any >> feedback. >> >> Regards, >> >> Shunping >> >
