Hi, I'm trying to redirect Spark logs to slf4j. Spark seem to be using Log4J, so I did the typical steps of forcing a Log4J-based framework to use slf4j - manually excluded slf4j-log4j12 and log4j, and included log4j-over-slf4j. When doing that however Spark starts failing on initialization with: java.lang.StackOverflowError at java.lang.ThreadLocal.access$400(ThreadLocal.java:72) at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:376) at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:261) at java.lang.ThreadLocal.get(ThreadLocal.java:146) at java.lang.StringCoding.deref(StringCoding.java:63) at java.lang.StringCoding.encode(StringCoding.java:330) at java.lang.String.getBytes(String.java:916) at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242) at java.io.File.exists(File.java:813) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080) at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047) at sun.misc.URLClassPath.findResource(URLClassPath.java:176) at java.net.URLClassLoader$2.run(URLClassLoader.java:551) at java.net.URLClassLoader$2.run(URLClassLoader.java:549) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findResource(URLClassLoader.java:548) at java.lang.ClassLoader.getResource(ClassLoader.java:1147) at org.apache.spark.Logging$class.initializeLogging(Logging.scala:109) at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:97) at org.apache.spark.Logging$class.log(Logging.scala:36) at org.apache.spark.util.Utils$.log(Utils.scala:47) <last 4 lines repeated many, many times>
There's some related work done in SPARK-1071<https://spark-project.atlassian.net/browse/SPARK-1071>, but it was resolved after 0.9.0 was released. In the last comment Sean refers to a StackOverflowError which was discussed in the mailing list, I assume it might be a problem similar to mine but I was not able to find that discussion. Is anyone aware of a way to redirect Spark 0.9.0 logs to slf4j? -- Best regards, Sergey Parhomenko
