[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023306#comment-16023306 ]
Glen Takahashi edited comment on SPARK-20871 at 5/24/17 5:46 PM: ----------------------------------------------------------------- Just attached an example heapdump of Janino logging adding gigabytes worth of strings to the heap was (Author: gtakaha...@palantir.com): An example of Janino logging adding gigabytes worth of strings to the heap > Only log Janino code in debug mode > ---------------------------------- > > Key: SPARK-20871 > URL: https://issues.apache.org/jira/browse/SPARK-20871 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.1 > Reporter: Glen Takahashi > Attachments: 6a57e344-3fcf-11e7-85cc-52a06df2a489.png > > > Currently if Janino code compilation fails, it will log the entirety of the > code in the executors. Because the generated code can often be very large, > the logging can cause heap pressure on the driver and cause it to fall over. > I propose removing the "$formatted" from here: > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L964 -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org