I think currently there's no API in Spark Streaming you can use to get the file names for file input streams. Actually it is not trivial to support this, may be you could file a JIRA with wishes you want the community to support, so anyone who is interested can take a crack on this.
Thanks Jerry 2015-04-29 0:13 GMT+08:00 lokeshkumar <lok...@dataken.net>: > Hi Forum, > > Using spark streaming and listening to the files in HDFS using > textFileStream/fileStream methods, how do we get the fileNames which are > read by these methods? > > I used textFileStream which has file contents in JavaDStream and I got no > success with fileStream as it is throwing me a compilation error with spark > version 1.3.1. > > Can someone please tell me if we have an API function or any other way to > get the file names that these streaming methods read? > > Thanks > Lokesh > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-textFileStream-fileStream-Get-file-name-tp22692.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
