[
https://issues.apache.org/jira/browse/SPARK-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu resolved SPARK-19718.
----------------------------------
Resolution: Fixed
Assignee: Shixiong Zhu
Fix Version/s: 2.2.0
> Fix flaky test:
> org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite:
> stress test for failOnDataLoss=false
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19718
> URL: https://issues.apache.org/jira/browse/SPARK-19718
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Shixiong Zhu
> Assignee: Shixiong Zhu
> Fix For: 2.2.0
>
>
> SPARK-19617 changed HDFSMetadataLog to enable interrupts when using the local
> file system. However, now we hit HADOOP-12074: `Shell.runCommand` converts
> `InterruptedException` to `new IOException(ie.toString())` before Hadoop 2.8.
> Test failure:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/2504/consoleFull
> {code}
> [info] - stress test for failOnDataLoss=false *** FAILED *** (1 minute, 1
> second)
> [info] org.apache.spark.sql.streaming.StreamingQueryException: Query [id =
> 27d45f4f-14dc-4c74-8b52-4bbd4f2b9bec, runId =
> 23b8c1ea-4da9-4096-967a-692933e4b319] terminated with exception:
> java.lang.InterruptedException
> [info] at
> org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:304)
> [info] at
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:190)
> [info] Cause: java.io.IOException: java.lang.InterruptedException
> [info] at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
> [info] at org.apache.hadoop.util.Shell.run(Shell.java:478)
> [info] at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766)
> [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:859)
> [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:842)
> [info] at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661)
> [info] at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:300)
> [info] at
> org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1014)
> [info] at
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
> [info] at
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(ChecksumFs.java:354)
> [info] at
> org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:394)
> [info] at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
> [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:680)
> [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
> [info] at
> org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> [info] at org.apache.hadoop.fs.FileContext.create(FileContext.java:676)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
