[
https://issues.apache.org/jira/browse/LOG4J2-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470640#comment-17470640
]
Ralph Goers commented on LOG4J2-2903:
-------------------------------------
Guys, this seems to be coming from Spark and Spark seems to be logging using
Log4j 1.x. Log4j 1.x has known issues in this area where Appenders call things
that perform logging which causes a StackOverflowError. From what I gather
Spark still has not moved to Log4j 2. See
https://issues.apache.org/jira/browse/SPARK-6305, which for some reason has a
status of resolved/fixed despite all the comments saying it hasn't been done
yet.
I am going to have to mark this as resolved since you aren't using Log4j 2. You
could try following some of the postings I found in a search to see if they
will help you to replace Log4j 1 with Log4j 2.
> Cannot log to HDFS with a custom appender?
> ------------------------------------------
>
> Key: LOG4J2-2903
> URL: https://issues.apache.org/jira/browse/LOG4J2-2903
> Project: Log4j 2
> Issue Type: Bug
> Components: Appenders, JMX, Scala API
> Affects Versions: 2.12.1
> Environment: Spark 2.4.6, Scala 2.1.2, Hadoop 3.2.1
> Unit tests: Windows 10, maven Eclipse 4.7 (Scala IDE) Junit 4
> Runtime tests: Ubuntu 16.04,18.04, (test/staging), Spark cluster, HDFS
> cluster
> Reporter: Thomas Humphries
> Priority: Minor
> Attachments: HDFSAppender.scala, LoggingTest.scala,
> MavenDependencies.txt, log4j.properties
>
>
> *Overview*
> [(x-post from public StackOverflow
> question)|https://stackoverflow.com/questions/63276838/scala-how-to-create-logs-on-hdfs-with-a-log4j-custom-appender]
> We want to log our Spark job activity using log4j to write log files to HDFS.
> - Java 8, Spark 2.4.6, Scala 2.1.2, Hadoop 3.2.1
> We were Unable to find a native apache log4j appenders to write to HDFS
> (Apache Flume in not an option), so set out to write our own.
> We constructed a simple custom log4j appender (Spark-submit with our
> java.jar writes to HDFS).
> *does not work*
> Scala logic matched Java; however, logging to HDFS fails consistently with a
> Stack overflow or OOM error.
> We cannot find anything obviously wrong?
> *The implementation:*
> Here is a simple hello-world project to reproduce the issue:
> - Scala project (Maven build), containing a simple main class, and:
> - HDFSAppender class (extends log4j AppenderSkeleton)
> - LoggingTest unit test class
> - log4j configuration file for testing
> **The error**
> Any logging event causes the program to crash with a stack overflow error:
> {code:bash}
> java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
> Caused by: java.lang.StackOverflowError
> at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
> at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
> at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
> {code}
> Then the following messages appear, implying some kind of loop/race condition:
> {code:bash}
> at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> at org.apache.log4j.Category.callAppenders(Category.java:206)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:401)
> at org.apache.spark.internal.Logging.logWarning(Logging.scala:66)
> at org.apache.spark.internal.Logging.logWarning$(Logging.scala:65)
> at org.apache.spark.SparkContext$.logWarning(SparkContext.scala:2442)
> at
> org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$5(SparkContext.scala:2500)
> at
> org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$5$adapted(SparkContext.scala:2491)
> at scala.Option.foreach(Option.scala:274)
> at
> org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2491)
> at
> org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2568)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:85)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
> at scala.Option.getOrElse(Option.scala:138)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
> at
> com.obfuscated.test.spark.log4j.HDFSLogWriter.write(HDFSLogWriter.scala:23)
> at
> com.obfuscated.test.spark.log4j.HDFSAppender.append(HDFSAppender.scala:63)
> (repeats 48 times...)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
