Github user srdo commented on the issue:
https://github.com/apache/storm/pull/2441
Thanks for explaining @revans2.
Please correct me if I get any of this wrong, I'm not very familiar with
Hadoop:
The issue is that you want the daemons to use HdfsBlobStore, which requires
you to put storm-hdfs on the daemon classpath, and additionally add the output
of `hadoop classpath` to the daemon classpath in order to load the Hadoop jars
and configuration files. For any topology code you'd just be handling excluding
slf4j-log4j12 through the usual Maven exclusion mechanism, but for the daemons
you can't easily remove slf4j-log4j12 because you'd have to filter `hadoop
classpath` somehow.
I don't like putting log4j-1.2-api on the classpath as a fix for this
because we'll lose whatever is logged to the log4j 1 interface. There is also a
second related issue that is not fixed by adding log4j-1.2-api; we'll have
multiple SLF4J bindings on the daemon classpath. Since Hadoop adds
slf4j-log4j12 (log4j 1) and Storm ships with log4j-slf4j-impl (log4j 2), we
can't be sure which logging implementation SLF4J will bind to (and where the
daemon log will go). Per the SLF4J documentation regarding multiple bindings:
> The warning emitted by SLF4J is just that, a warning. Even when multiple
bindings are present, SLF4J will pick one logging framework/implementation and
bind with it. The way SLF4J picks a binding is determined by the JVM and for
all practical purposes should be considered random. As of version 1.6.6, SLF4J
will name the framework/implementation class it is actually bound to.
It seems someone else encountered this issue at
https://issues.apache.org/jira/browse/HADOOP-13344. It might make sense to
lobby a bit to get that feature added. If we're going to add log4j-1.2-api to
the daemon classpath I'd like it to be only temporarily until the linked issue
makes it in, and then log4j-1.2-api would probably need to be only on the
daemon classpath and not in any of the external modules or the worker classpath
to avoid interfering with topologies that use log4j-over-slf4j.
An alternative temporary fix would be writing a script or program that
could expand the Hadoop classpath wildcards that contain slf4j-log4j12* and
remove that jar? That way we could provide the functionality of
https://issues.apache.org/jira/browse/HADOOP-13344 ourselves until that feature
makes it in.
---
