[jira] [Commented] (SPARK-11170) EOFException on History server reading in progress lz4
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100537#comment-16100537 ] Eric Vandenberg commented on SPARK-11170: - There's a fix for this, see https://issues.apache.org/jira/browse/SPARK-21447 it's been reviewed/tested and pending merge. > EOFException on History server reading in progress lz4 > > > Key: SPARK-11170 > URL: https://issues.apache.org/jira/browse/SPARK-11170 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.5.1 > Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) > Spark: 1.5.x (c27e1904) >Reporter: Sebastian YEPES FERNANDEZ > > The Spark History server is not able to read/save the jobs history if Spark > is configured to use > "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it > continuously generated the following error: > {code} > ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Exception encountered when attempting to load application log > hdfs://DATA/user/spark/his > tory/application_1444297190346_0073_1.lz4.inprogress > java.io.EOFException: Stream ended prematurely > at > net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) > at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) > at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > at java.io.InputStreamReader.read(InputStreamReader.java:184) > at java.io.BufferedReader.fill(BufferedReader.java:161) > at java.io.BufferedReader.readLine(BufferedReader.java:324) > at java.io.BufferedReader.readLine(BufferedReader.java:389) > at > scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) > at > org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Replaying log path: > hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i > nprogress > {code} > As a workaround setting > "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" > makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11170) EOFException on History server reading in progress lz4
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054701#comment-16054701 ] remoteServer commented on SPARK-11170: -- Do we have steps to reproduce the issue? I have enabled spark event log compression, lz4 files are generated, however, I do not see any error message in the History Server logs. I am in Spark 2.1.0 > EOFException on History server reading in progress lz4 > > > Key: SPARK-11170 > URL: https://issues.apache.org/jira/browse/SPARK-11170 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.5.1 > Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) > Spark: 1.5.x (c27e1904) >Reporter: Sebastian YEPES FERNANDEZ > > The Spark History server is not able to read/save the jobs history if Spark > is configured to use > "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it > continuously generated the following error: > {code} > ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Exception encountered when attempting to load application log > hdfs://DATA/user/spark/his > tory/application_1444297190346_0073_1.lz4.inprogress > java.io.EOFException: Stream ended prematurely > at > net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) > at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) > at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > at java.io.InputStreamReader.read(InputStreamReader.java:184) > at java.io.BufferedReader.fill(BufferedReader.java:161) > at java.io.BufferedReader.readLine(BufferedReader.java:324) > at java.io.BufferedReader.readLine(BufferedReader.java:389) > at > scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) > at > org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Replaying log path: > hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i > nprogress > {code} > As a workaround setting > "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" > makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11170) EOFException on History server reading in progress lz4
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962102#comment-14962102 ] Josh Rosen commented on SPARK-11170: [~vanzin], why do we use the IO compression codec when writing history server event logs? This seems to cause a lot of code complexity and hassle, since we need to embed the compression codec class into the file. Instead, why not just settle on always using GZip for these files? > EOFException on History server reading in progress lz4 > > > Key: SPARK-11170 > URL: https://issues.apache.org/jira/browse/SPARK-11170 > Project: Spark > Issue Type: Bug > Components: Web UI, YARN >Affects Versions: 1.5.1 > Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) > Spark: 1.5.x (c27e1904) >Reporter: Sebastian YEPES FERNANDEZ > > The Spark History server is not able to read/save the jobs history if Spark > is configured to use > "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it > continuously generated the following error: > {code} > ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Exception encountered when attempting to load application log > hdfs://DATA/user/spark/his > tory/application_1444297190346_0073_1.lz4.inprogress > java.io.EOFException: Stream ended prematurely > at > net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) > at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) > at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > at java.io.InputStreamReader.read(InputStreamReader.java:184) > at java.io.BufferedReader.fill(BufferedReader.java:161) > at java.io.BufferedReader.readLine(BufferedReader.java:324) > at java.io.BufferedReader.readLine(BufferedReader.java:389) > at > scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) > at > org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Replaying log path: > hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i > nprogress > {code} > As a workaround setting > "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" > makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11170) EOFException on History server reading in progress lz4
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962104#comment-14962104 ] Marcelo Vanzin commented on SPARK-11170: It's not enabled by default ({{spark.eventLog.compress}} controls it), but if enabled that uses whatever is the configured codec. I didn't write that feature, but I'm not against just sticking to one hardcoded coded. > EOFException on History server reading in progress lz4 > > > Key: SPARK-11170 > URL: https://issues.apache.org/jira/browse/SPARK-11170 > Project: Spark > Issue Type: Bug > Components: Web UI, YARN >Affects Versions: 1.5.1 > Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) > Spark: 1.5.x (c27e1904) >Reporter: Sebastian YEPES FERNANDEZ > > The Spark History server is not able to read/save the jobs history if Spark > is configured to use > "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it > continuously generated the following error: > {code} > ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Exception encountered when attempting to load application log > hdfs://DATA/user/spark/his > tory/application_1444297190346_0073_1.lz4.inprogress > java.io.EOFException: Stream ended prematurely > at > net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) > at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) > at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > at java.io.InputStreamReader.read(InputStreamReader.java:184) > at java.io.BufferedReader.fill(BufferedReader.java:161) > at java.io.BufferedReader.readLine(BufferedReader.java:324) > at java.io.BufferedReader.readLine(BufferedReader.java:389) > at > scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) > at > org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Replaying log path: > hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i > nprogress > {code} > As a workaround setting > "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" > makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11170) EOFException on History server reading in progress lz4
[ https://issues.apache.org/jira/browse/SPARK-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962107#comment-14962107 ] Sebastian YEPES FERNANDEZ commented on SPARK-11170: --- I also tried 'spark.eventLog.compress = false' as a workaround, but it was still saving the file with the IO codec. > EOFException on History server reading in progress lz4 > > > Key: SPARK-11170 > URL: https://issues.apache.org/jira/browse/SPARK-11170 > Project: Spark > Issue Type: Bug > Components: Web UI, YARN >Affects Versions: 1.5.1 > Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) > Spark: 1.5.x (c27e1904) >Reporter: Sebastian YEPES FERNANDEZ > > The Spark History server is not able to read/save the jobs history if Spark > is configured to use > "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it > continuously generated the following error: > {code} > ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Exception encountered when attempting to load application log > hdfs://DATA/user/spark/his > tory/application_1444297190346_0073_1.lz4.inprogress > java.io.EOFException: Stream ended prematurely > at > net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) > at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) > at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) > at java.io.InputStreamReader.read(InputStreamReader.java:184) > at java.io.BufferedReader.fill(BufferedReader.java:161) > at java.io.BufferedReader.readLine(BufferedReader.java:324) > at java.io.BufferedReader.readLine(BufferedReader.java:389) > at > scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) > at > org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) > at > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: > Replaying log path: > hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i > nprogress > {code} > As a workaround setting > "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" > makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org