To: Akhil Das
CC: bit1...@163.com; Vadim Bichutskiy; lokeshkumar; user
Subject: Re: Re: Spark streaming - textFileStream/fileStream - Get file name
Yes, looks like a solution but quite tricky. You have to parse the debug string
to get the file name, also relies on HadoopRDD to get the file name
sai.sai.s...@gmail.com
*Date:* 2015-04-29 10:54
*To:* Vadim Bichutskiy vadim.bichuts...@gmail.com
*CC:* bit1...@163.com; lokeshkumar lok...@dataken.net; user
user@spark.apache.org
*Subject:* Re: Re: Spark streaming - textFileStream/fileStream - Get
file name
I think it might be useful
@spark.apache.org
*Subject:* Re: Spark streaming - textFileStream/fileStream - Get file
name
I think currently there's no API in Spark Streaming you can use to get
the file names for file input streams. Actually it is not trivial to
support this, may be you could file a JIRA with wishes you want
I think currently there's no API in Spark Streaming you can use to get the
file names for file input streams. Actually it is not trivial to support
this, may be you could file a JIRA with wishes you want the community to
support, so anyone who is interested can take a crack on this.
Thanks
Jerry
: Re: Spark streaming - textFileStream/fileStream - Get file name
I think currently there's no API in Spark Streaming you can use to get the file
names for file input streams. Actually it is not trivial to support this, may
be you could file a JIRA with wishes you want the community to support, so
the data in RDD is from
--
bit1...@163.com
*From:* Saisai Shao sai.sai.s...@gmail.com
*Date:* 2015-04-29 10:10
*To:* lokeshkumar lok...@dataken.net
*CC:* spark users user@spark.apache.org
*Subject:* Re: Spark streaming - textFileStream/fileStream - Get file name
I
streaming - textFileStream/fileStream - Get file
name
I think currently there's no API in Spark Streaming you can use to get
the file names for file input streams. Actually it is not trivial to
support this, may be you could file a JIRA with wishes you want the
community to support, so anyone who
or
SparkContext.wholeTextFile, there is no way in RDD to figure out the file
information where the data in RDD is from
bit1...@163.com
From: Saisai Shao
Date: 2015-04-29 10:10
To: lokeshkumar
CC: spark users
Subject: Re: Spark streaming - textFileStream/fileStream - Get file name
I think
