[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated SPARK-19407:
---------------------------------
Target Version/s: 2.1.1, 2.2.0
> defaultFS is used FileSystem.get instead of getting it from uri scheme
> ----------------------------------------------------------------------
>
> Key: SPARK-19407
> URL: https://issues.apache.org/jira/browse/SPARK-19407
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Amit Assudani
> Labels: checkpoint, filesystem, starter, streaming
>
> Caused by: java.lang.IllegalArgumentException: Wrong FS:
> s3a://**************/checkpoint/7b2231a3-d845-4740-bfa3-681850e5987f/metadata,
> expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:649)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
> at
> org.apache.spark.sql.execution.streaming.StreamMetadata$.read(StreamMetadata.scala:51)
> at
> org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:100)
> at
> org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:232)
> at
> org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:269)
> at
> org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:262)
> Can easily replicate on spark standalone cluster by providing checkpoint
> location uri scheme anything other than "file://" and not overriding in
> config.
> WorkAround --conf spark.hadoop.fs.defaultFS=s3a://somebucket or set it in
> sparkConf or spark-default.conf
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
