To: Akhil Das
CC: bit1...@163.com; Vadim Bichutskiy; lokeshkumar; user
Subject: Re: Re: Spark streaming - textFileStream/fileStream - Get file name
Yes, looks like a solution but quite tricky. You have to parse the debug string
to get the file name, also relies on HadoopRDD to get the file name
sai.sai.s...@gmail.com
*Date:* 2015-04-29 10:54
*To:* Vadim Bichutskiy vadim.bichuts...@gmail.com
*CC:* bit1...@163.com; lokeshkumar lok...@dataken.net; user
user@spark.apache.org
*Subject:* Re: Re: Spark streaming - textFileStream/fileStream - Get
file name
I think it might be useful
@spark.apache.org
*Subject:* Re: Spark streaming - textFileStream/fileStream - Get file
name
I think currently there's no API in Spark Streaming you can use to get
the file names for file input streams. Actually it is not trivial to
support this, may be you could file a JIRA with wishes you want
://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-tp22692.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr
: Re: Spark streaming - textFileStream/fileStream - Get file name
I think currently there's no API in Spark Streaming you can use to get the file
names for file input streams. Actually it is not trivial to support this, may
be you could file a JIRA with wishes you want the community to support, so
the data in RDD is from
--
bit1...@163.com
*From:* Saisai Shao sai.sai.s...@gmail.com
*Date:* 2015-04-29 10:10
*To:* lokeshkumar lok...@dataken.net
*CC:* spark users user@spark.apache.org
*Subject:* Re: Spark streaming - textFileStream/fileStream - Get file name
I
Lokesh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-tp22692.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
: Spark streaming - textFileStream/fileStream - Get file name
I think it might be useful in Spark Streaming's file input stream, but not sure
is it useful in SparkContext#textFile, since we specify the file by our own, so
why we still need to know the file name.
I will open up a JIRA to mention
-textFileStream-fileStream-Get-file-name-tp22692.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
