Ivan Sharamet created SPARK-21996:
-------------------------------------
Summary: Streaming ignores files with spaces in the file names
Key: SPARK-21996
URL: https://issues.apache.org/jira/browse/SPARK-21996
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Environment: openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
Reporter: Ivan Sharamet
I tried to stream text files from folder I noticed that files with spaces in
the name ignored and there are warnings in the log:
{code:bash}
17/09/13 16:15:14 WARN InMemoryFileIndex: The directory file:/dir/my%20file.txt
was not found. Was it deleted very recently?
{code}
I found that this happens to the duplicate file path URI encoding and actual
URI inside path objects looks like this {{file:/dir/my%2520file.txt}}.
To reproduce this issue just place some text files with spaces in their names
and run simple streaming code:
{code:scala}
sparkSession.readStream.textFile("/in")
.writeStream
.option("checkpointLocation", "/checkpoint")
.start("/out")
.awaitTermination()
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
