[jira] [Commented] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209425#comment-17209425 ] Sachin Pasalkar commented on SPARK-30542: - [~kabhwan] Can't we make this configurable? > Two Spark structured streaming jobs cannot write to same base path > -- > > Key: SPARK-30542 > URL: https://issues.apache.org/jira/browse/SPARK-30542 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Sivakumar >Priority: Major > > Hi All, > Spark Structured Streaming doesn't allow two structured streaming jobs to > write data to the same base directory which is possible with using dstreams. > As __spark___metadata directory will be created by default for one job, > second job cannot use the same directory as base path as already > _spark__metadata directory is created by other job, It is throwing exception. > Is there any workaround for this, other than creating separate base path's > for both the jobs. > Is it possible to create the __spark__metadata directory else where or > disable without any data loss. > If I had to change the base path for both the jobs, then my whole framework > will get impacted, So i don't want to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209297#comment-17209297 ] Sachin Pasalkar commented on SPARK-30542: - [~SparkSiva] Did you get a response to it? I see it's a bug in the latest release as well [https://github.com/apache/spark/blob/5472170a2b35864c617bdb846ff7123533765a16/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala#L36] I see a hardcoded value which bounds to fail for multiple jobs writing to same path > Two Spark structured streaming jobs cannot write to same base path > -- > > Key: SPARK-30542 > URL: https://issues.apache.org/jira/browse/SPARK-30542 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Sivakumar >Priority: Major > > Hi All, > Spark Structured Streaming doesn't allow two structured streaming jobs to > write data to the same base directory which is possible with using dstreams. > As __spark___metadata directory will be created by default for one job, > second job cannot use the same directory as base path as already > _spark__metadata directory is created by other job, It is throwing exception. > Is there any workaround for this, other than creating separate base path's > for both the jobs. > Is it possible to create the __spark__metadata directory else where or > disable without any data loss. > If I had to change the base path for both the jobs, then my whole framework > will get impacted, So i don't want to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30460) Spark checkpoint failing after some run with S3 path
[ https://issues.apache.org/jira/browse/SPARK-30460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012784#comment-17012784 ] Sachin Pasalkar edited comment on SPARK-30460 at 1/10/20 12:19 PM: --- [~gsomogyi] Yes may be or may be not. I was able to run this on my production for 4-6 hours without any other issues for 4-5 times. It always failed with this issue. If this fix the some part of problem we should fix it. I understand spark 3.0 has new committer but as you said it is not deeply tested. Soon I am going to run my Production with this fix in place, I will update ticket around next EOW. If I was able to run system smoothly or not was (Author: sachin): Yes may be or may be not. I was able to run this on my production for 4-6 hours without any other issues for 4-5 times. It always failed with this issue. If this fix the some part of problem we should fix it. I understand spark 3.0 has new committer but as you said it is not deeply tested. Soon I am going to run my Production with this fix in place, I will update ticket around next EOW. If I was able to run system smoothly or not > Spark checkpoint failing after some run with S3 path > - > > Key: SPARK-30460 > URL: https://issues.apache.org/jira/browse/SPARK-30460 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.4.4 >Reporter: Sachin Pasalkar >Priority: Major > > We are using EMR with the SQS as source of stream. However it is failing, > after 4-6 hours of run, with below exception. Application shows its running > but stops the processing the messages > {code:java} > 2020-01-06 13:04:10,548 WARN [BatchedWriteAheadLog Writer] > org.apache.spark.streaming.util.BatchedWriteAheadLog:BatchedWriteAheadLog > Writer failed to write ArrayBuffer(Record(java.nio.HeapByteBuffer[pos=0 > lim=1226 cap=1226],1578315850302,Future())) > java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) > at > org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:50) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.org$apache$spark$streaming$util$BatchedWriteAheadLog$$flushRecords(BatchedWriteAheadLog.scala:175) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog$$anon$1.run(BatchedWriteAheadLog.scala:142) > at java.lang.Thread.run(Thread.java:748) > 2020-01-06 13:04:10,554 WARN [wal-batching-thread-pool-0] > org.apache.spark.streaming.scheduler.ReceivedBlockTracker:Exception thrown > while writing record: > BlockAdditionEvent(ReceivedBlockInfo(0,Some(3),None,WriteAheadLogBasedStoreResult(input-0-1578315849800,Some(3),FileBasedWriteAheadLogSegment(s3://mss-prod-us-east-1-ueba-bucket/streaming/checkpoint/receivedData/0/log-1578315850001-1578315910001,0,5175 > to the WriteAheadLog. > org.apache.spark.SparkException: Exception thrown in awaitResult: > at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.write(BatchedWriteAheadLog.scala:84) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.writeToLog(ReceivedBlockTracker.scala:242) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.addBlock(ReceivedBlockTracker.scala:89) > at > org.apache.spark.streaming.scheduler.ReceiverTracker.org$apache$spark$streaming$scheduler$ReceiverTracker$$addBlock(ReceiverTracker.scala:347) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1$$anonfun$run$1.apply$mcV$sp(ReceiverTracker.scala:522) > at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$
[jira] [Commented] (SPARK-30460) Spark checkpoint failing after some run with S3 path
[ https://issues.apache.org/jira/browse/SPARK-30460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012784#comment-17012784 ] Sachin Pasalkar commented on SPARK-30460: - Yes may be or may be not. I was able to run this on my production for 4-6 hours without any other issues for 4-5 times. It always failed with this issue. If this fix the some part of problem we should fix it. I understand spark 3.0 has new committer but as you said it is not deeply tested. Soon I am going to run my Production with this fix in place, I will update ticket around next EOW. If I was able to run system smoothly or not > Spark checkpoint failing after some run with S3 path > - > > Key: SPARK-30460 > URL: https://issues.apache.org/jira/browse/SPARK-30460 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.4.4 >Reporter: Sachin Pasalkar >Priority: Major > > We are using EMR with the SQS as source of stream. However it is failing, > after 4-6 hours of run, with below exception. Application shows its running > but stops the processing the messages > {code:java} > 2020-01-06 13:04:10,548 WARN [BatchedWriteAheadLog Writer] > org.apache.spark.streaming.util.BatchedWriteAheadLog:BatchedWriteAheadLog > Writer failed to write ArrayBuffer(Record(java.nio.HeapByteBuffer[pos=0 > lim=1226 cap=1226],1578315850302,Future())) > java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) > at > org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:50) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.org$apache$spark$streaming$util$BatchedWriteAheadLog$$flushRecords(BatchedWriteAheadLog.scala:175) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog$$anon$1.run(BatchedWriteAheadLog.scala:142) > at java.lang.Thread.run(Thread.java:748) > 2020-01-06 13:04:10,554 WARN [wal-batching-thread-pool-0] > org.apache.spark.streaming.scheduler.ReceivedBlockTracker:Exception thrown > while writing record: > BlockAdditionEvent(ReceivedBlockInfo(0,Some(3),None,WriteAheadLogBasedStoreResult(input-0-1578315849800,Some(3),FileBasedWriteAheadLogSegment(s3://mss-prod-us-east-1-ueba-bucket/streaming/checkpoint/receivedData/0/log-1578315850001-1578315910001,0,5175 > to the WriteAheadLog. > org.apache.spark.SparkException: Exception thrown in awaitResult: > at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.write(BatchedWriteAheadLog.scala:84) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.writeToLog(ReceivedBlockTracker.scala:242) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.addBlock(ReceivedBlockTracker.scala:89) > at > org.apache.spark.streaming.scheduler.ReceiverTracker.org$apache$spark$streaming$scheduler$ReceiverTracker$$addBlock(ReceiverTracker.scala:347) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1$$anonfun$run$1.apply$mcV$sp(ReceiverTracker.scala:522) > at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1.run(ReceiverTracker.scala:520) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazo
[jira] [Commented] (SPARK-30460) Spark checkpoint failing after some run with S3 path
[ https://issues.apache.org/jira/browse/SPARK-30460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012728#comment-17012728 ] Sachin Pasalkar commented on SPARK-30460: - [~gsomogyi] Yes I am using S3 for checkpoint and as we know S3 do not support appending object. However, if you look at the exception stack-trace, it seems it is trying to append the object, which causing failure. If you follow the stack trace `FileBasedWriteAheadLogWriter` gets `outputstream` using HDFSUtils. However HDFSUtils, only supports case for HDFS not for the other non append-able system. I don't see it as issue of consistency model but bug in code > Spark checkpoint failing after some run with S3 path > - > > Key: SPARK-30460 > URL: https://issues.apache.org/jira/browse/SPARK-30460 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.4.4 >Reporter: Sachin Pasalkar >Priority: Major > > We are using EMR with the SQS as source of stream. However it is failing, > after 4-6 hours of run, with below exception. Application shows its running > but stops the processing the messages > {code:java} > 2020-01-06 13:04:10,548 WARN [BatchedWriteAheadLog Writer] > org.apache.spark.streaming.util.BatchedWriteAheadLog:BatchedWriteAheadLog > Writer failed to write ArrayBuffer(Record(java.nio.HeapByteBuffer[pos=0 > lim=1226 cap=1226],1578315850302,Future())) > java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) > at > org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:50) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.org$apache$spark$streaming$util$BatchedWriteAheadLog$$flushRecords(BatchedWriteAheadLog.scala:175) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog$$anon$1.run(BatchedWriteAheadLog.scala:142) > at java.lang.Thread.run(Thread.java:748) > 2020-01-06 13:04:10,554 WARN [wal-batching-thread-pool-0] > org.apache.spark.streaming.scheduler.ReceivedBlockTracker:Exception thrown > while writing record: > BlockAdditionEvent(ReceivedBlockInfo(0,Some(3),None,WriteAheadLogBasedStoreResult(input-0-1578315849800,Some(3),FileBasedWriteAheadLogSegment(s3://mss-prod-us-east-1-ueba-bucket/streaming/checkpoint/receivedData/0/log-1578315850001-1578315910001,0,5175 > to the WriteAheadLog. > org.apache.spark.SparkException: Exception thrown in awaitResult: > at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.write(BatchedWriteAheadLog.scala:84) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.writeToLog(ReceivedBlockTracker.scala:242) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.addBlock(ReceivedBlockTracker.scala:89) > at > org.apache.spark.streaming.scheduler.ReceiverTracker.org$apache$spark$streaming$scheduler$ReceiverTracker$$addBlock(ReceiverTracker.scala:347) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1$$anonfun$run$1.apply$mcV$sp(ReceiverTracker.scala:522) > at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1.run(ReceiverTracker.scala:520) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:118
[jira] [Commented] (SPARK-30460) Spark checkpoint failing after some run with S3 path
[ https://issues.apache.org/jira/browse/SPARK-30460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010742#comment-17010742 ] Sachin Pasalkar commented on SPARK-30460: - I had a long discussion with AWS folks but they are asking to report this to open source to verify it > Spark checkpoint failing after some run with S3 path > - > > Key: SPARK-30460 > URL: https://issues.apache.org/jira/browse/SPARK-30460 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.4.4 >Reporter: Sachin Pasalkar >Priority: Major > > We are using EMR with the SQS as source of stream. However it is failing, > after 4-6 hours of run, with below exception. Application shows its running > but stops the processing the messages > {code:java} > 2020-01-06 13:04:10,548 WARN [BatchedWriteAheadLog Writer] > org.apache.spark.streaming.util.BatchedWriteAheadLog:BatchedWriteAheadLog > Writer failed to write ArrayBuffer(Record(java.nio.HeapByteBuffer[pos=0 > lim=1226 cap=1226],1578315850302,Future())) > java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) > at > org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:50) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.org$apache$spark$streaming$util$BatchedWriteAheadLog$$flushRecords(BatchedWriteAheadLog.scala:175) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog$$anon$1.run(BatchedWriteAheadLog.scala:142) > at java.lang.Thread.run(Thread.java:748) > 2020-01-06 13:04:10,554 WARN [wal-batching-thread-pool-0] > org.apache.spark.streaming.scheduler.ReceivedBlockTracker:Exception thrown > while writing record: > BlockAdditionEvent(ReceivedBlockInfo(0,Some(3),None,WriteAheadLogBasedStoreResult(input-0-1578315849800,Some(3),FileBasedWriteAheadLogSegment(s3://mss-prod-us-east-1-ueba-bucket/streaming/checkpoint/receivedData/0/log-1578315850001-1578315910001,0,5175 > to the WriteAheadLog. > org.apache.spark.SparkException: Exception thrown in awaitResult: > at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226) > at > org.apache.spark.streaming.util.BatchedWriteAheadLog.write(BatchedWriteAheadLog.scala:84) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.writeToLog(ReceivedBlockTracker.scala:242) > at > org.apache.spark.streaming.scheduler.ReceivedBlockTracker.addBlock(ReceivedBlockTracker.scala:89) > at > org.apache.spark.streaming.scheduler.ReceiverTracker.org$apache$spark$streaming$scheduler$ReceiverTracker$$addBlock(ReceiverTracker.scala:347) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1$$anonfun$run$1.apply$mcV$sp(ReceiverTracker.scala:522) > at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1.run(ReceiverTracker.scala:520) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.UnsupportedOperationException > at > com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) > at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) > at > com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) > at > org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) > at > org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) > at > org.apache.spark.streaming.util.File
[jira] [Created] (SPARK-30460) Spark checkpoint failing after some run with S3 path
Sachin Pasalkar created SPARK-30460: --- Summary: Spark checkpoint failing after some run with S3 path Key: SPARK-30460 URL: https://issues.apache.org/jira/browse/SPARK-30460 Project: Spark Issue Type: Bug Components: DStreams Affects Versions: 2.4.4 Reporter: Sachin Pasalkar We are using EMR with the SQS as source of stream. However it is failing, after 4-6 hours of run, with below exception. Application shows its running but stops the processing the messages {code:java} 2020-01-06 13:04:10,548 WARN [BatchedWriteAheadLog Writer] org.apache.spark.streaming.util.BatchedWriteAheadLog:BatchedWriteAheadLog Writer failed to write ArrayBuffer(Record(java.nio.HeapByteBuffer[pos=0 lim=1226 cap=1226],1578315850302,Future())) java.lang.UnsupportedOperationException at com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) at org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) at org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) at org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) at org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:50) at org.apache.spark.streaming.util.BatchedWriteAheadLog.org$apache$spark$streaming$util$BatchedWriteAheadLog$$flushRecords(BatchedWriteAheadLog.scala:175) at org.apache.spark.streaming.util.BatchedWriteAheadLog$$anon$1.run(BatchedWriteAheadLog.scala:142) at java.lang.Thread.run(Thread.java:748) 2020-01-06 13:04:10,554 WARN [wal-batching-thread-pool-0] org.apache.spark.streaming.scheduler.ReceivedBlockTracker:Exception thrown while writing record: BlockAdditionEvent(ReceivedBlockInfo(0,Some(3),None,WriteAheadLogBasedStoreResult(input-0-1578315849800,Some(3),FileBasedWriteAheadLogSegment(s3://mss-prod-us-east-1-ueba-bucket/streaming/checkpoint/receivedData/0/log-1578315850001-1578315910001,0,5175 to the WriteAheadLog. org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226) at org.apache.spark.streaming.util.BatchedWriteAheadLog.write(BatchedWriteAheadLog.scala:84) at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.writeToLog(ReceivedBlockTracker.scala:242) at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.addBlock(ReceivedBlockTracker.scala:89) at org.apache.spark.streaming.scheduler.ReceiverTracker.org$apache$spark$streaming$scheduler$ReceiverTracker$$addBlock(ReceiverTracker.scala:347) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1$$anonfun$run$1.apply$mcV$sp(ReceiverTracker.scala:522) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$receiveAndReply$1$$anon$1.run(ReceiverTracker.scala:520) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.UnsupportedOperationException at com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:150) at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:295) at org.apache.spark.streaming.util.HdfsUtils$.getOutputStream(HdfsUtils.scala:35) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream$lzycompute(FileBasedWriteAheadLogWriter.scala:32) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.stream(FileBasedWriteAheadLogWriter.scala:32) at org.apache.spark.streaming.util.FileBasedWriteAheadLogWriter.(FileBasedWriteAheadLogWriter.scala:35) at org.apache.spark.streaming.util.FileBasedWriteAheadLog.getLogWriter(FileBasedWriteAheadLog.scala:229) at org.apache.spark.streaming.util.FileBasedWriteAheadLog.write(FileBasedWriteAheadLog.scala:94) at org.apache.spark.streaming