[ https://issues.apache.org/jira/browse/SPARK-30599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020465#comment-17020465 ]
Maxim Gekk commented on SPARK-30599: ------------------------------------ I see at least 2 solutions: 1. Increase the limit from 100 to something bigger number. For example, 1000 2. Pass to LogAppender's constructor a sequence of strings, and filter incoming log events by the strings in the `append()` method. [~cloud_fan] [~dongjoon] [~hyukjin.kwon] [~maropu] [~srowen] WDYT? > SparkFunSuite.LogAppender throws java.lang.IllegalStateException > ---------------------------------------------------------------- > > Key: SPARK-30599 > URL: https://issues.apache.org/jira/browse/SPARK-30599 > Project: Spark > Issue Type: Test > Components: Spark Core, SQL, Tests > Affects Versions: 3.0.0 > Reporter: Maxim Gekk > Priority: Minor > Attachments: fail_trace.txt, success_trace.txt > > > The LogAppender has a limit for the number of logged event. By default, it is > 100. All tests that use LogAppender finish successfully with this limit but > sometime the test fail with the exception like: > {code:java} > java.lang.IllegalStateException: Number of events reached the limit of 100 > while logging CSV header matches to schema w/ enforceSchema. > at > org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200) > at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) > at > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) > at org.apache.log4j.Category.callAppenders(Category.java:206) > at org.apache.log4j.Category.forcedLog(Category.java:391) > at org.apache.log4j.Category.log(Category.java:856) > {code} > For example, the CSV test "SPARK-23786: warning should be printed if CSV > header doesn't conform to schema" uses 2 log appenders. The > `success_trace.txt` contains stored log event in normal runs, another one > `fail_trace.txt` I got while re-running the test in a loop. > The difference is in: > {code} > [35] Block broadcast_222 stored as values in memory (estimated size 286.9 > KiB, free 1998.5 MiB) > [36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2003.8 MiB) > [37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2003.9 MiB) > [38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2003.9 MiB) > [39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2003.9 MiB) > [40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2003.9 MiB) > [41] Removed broadcast_194_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2003.9 MiB) > [42] Removed broadcast_216_piece0 on 192.168.1.66:52354 in memory (size: 53.3 > KiB, free: 2004.0 MiB) > [43] Removed broadcast_195_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.0 MiB) > [44] Removed broadcast_219_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2004.0 MiB) > [45] Removed broadcast_198_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.0 MiB) > [46] Removed broadcast_201_piece0 on 192.168.1.66:52354 in memory (size: 53.4 > KiB, free: 2004.0 MiB) > [47] Removed broadcast_205_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.0 MiB) > [48] Removed broadcast_197_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.1 MiB) > [49] Removed broadcast_189_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2004.1 MiB) > [50] Removed broadcast_193_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.1 MiB) > [51] Removed broadcast_196_piece0 on 192.168.1.66:52354 in memory (size: 53.3 > KiB, free: 2004.2 MiB) > [52] Removed broadcast_206_piece0 on 192.168.1.66:52354 in memory (size: 53.4 > KiB, free: 2004.2 MiB) > [53] Removed broadcast_209_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2004.2 MiB) > [54] Removed broadcast_192_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.2 MiB) > [55] Removed broadcast_215_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.2 MiB) > [56] Removed broadcast_210_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.2 MiB) > [57] Removed broadcast_199_piece0 on 192.168.1.66:52354 in memory (size: 5.7 > KiB, free: 2004.2 MiB) > [58] Removed broadcast_221_piece0 on 192.168.1.66:52354 in memory (size: 53.4 > KiB, free: 2004.3 MiB) > [59] Removed broadcast_213_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.3 MiB) > [60] Removed broadcast_191_piece0 on 192.168.1.66:52354 in memory (size: 53.4 > KiB, free: 2004.4 MiB) > [61] Removed broadcast_220_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.4 MiB) > [62] Block broadcast_222_piece0 stored as bytes in memory (estimated size > 24.2 KiB, free 2002.0 MiB) > [63] Added broadcast_222_piece0 in memory on 192.168.1.66:52354 (size: 24.2 > KiB, free: 2004.4 MiB) > [64] Removed broadcast_203_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.4 MiB) > [65] Created broadcast 222 from apply at OutcomeOf.scala:85 > [66] Removed broadcast_218_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.4 MiB) > [67] Removed broadcast_211_piece0 on 192.168.1.66:52354 in memory (size: 53.4 > KiB, free: 2004.5 MiB) > [68] Removed broadcast_190_piece0 on 192.168.1.66:52354 in memory (size: 3.7 > KiB, free: 2004.5 MiB) > [69] Block broadcast_223 stored as values in memory (estimated size 286.9 > KiB, free 2002.5 MiB) > [70] Removed broadcast_212_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.5 MiB) > [71] Removed broadcast_188_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.5 MiB) > [72] Removed broadcast_187_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.5 MiB) > [73] Removed broadcast_202_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.6 MiB) > [74] Removed broadcast_217_piece0 on 192.168.1.66:52354 in memory (size: 24.2 > KiB, free: 2004.6 MiB) > [75] Block broadcast_223_piece0 stored as bytes in memory (estimated size > 24.2 KiB, free 2004.0 MiB) > [76] Added broadcast_223_piece0 in memory on 192.168.1.66:52354 (size: 24.2 > KiB, free: 2004.6 MiB) > [77] Created broadcast 223 from apply at OutcomeOf.scala:85 > [78] Starting job: apply at OutcomeOf.scala:85 > [79] Got job 134 (apply at OutcomeOf.scala:85) with 1 output partitions > [80] Final stage: ResultStage 134 (apply at OutcomeOf.scala:85) > [81] Parents of final stage: List() > [82] Missing parents: List() > [83] Submitting ResultStage 134 (MapPartitionsRDD[809] at apply at > OutcomeOf.scala:85), which has no missing parents > [84] Block broadcast_224 stored as values in memory (estimated size 11.0 KiB, > free 2004.0 MiB) > [85] Block broadcast_224_piece0 stored as bytes in memory (estimated size 5.7 > KiB, free 2004.0 MiB) > [86] Added broadcast_224_piece0 in memory on 192.168.1.66:52354 (size: 5.7 > KiB, free: 2004.5 MiB) > [87] Created broadcast 224 from broadcast at DAGScheduler.scala:1200 > [88] Submitting 1 missing tasks from ResultStage 134 (MapPartitionsRDD[809] > at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0)) > [89] Adding task set 134.0 with 1 tasks > [90] Starting task 0.0 in stage 134.0 (TID 134, 192.168.1.66, executor > driver, partition 0, PROCESS_LOCAL, 8029 bytes) > [91] Running task 0.0 in stage 134.0 (TID 134) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org