What do you mean by not detected? may be you forgot to trigger some action on the stream to get it executed. Like:
val list_join_action_stream = ssc.fileStream[LongWritable, Text, TextInputFormat](gc.input_dir, (t: Path) => true, false).map(_._2.toString) *list_join_action_stream.count().print()* Thanks Best Regards On Wed, May 13, 2015 at 7:18 PM, hotdog <[email protected]> wrote: > in spark streaming, I want to use fileStream to monitor a directory. But > the > files in that directory are compressed using lz4. So the new lz4 files are > not detected by the following code. How to detect these new files? > > val list_join_action_stream = ssc.fileStream[LongWritable, Text, > TextInputFormat](gc.input_dir, (t: Path) => true, false).map(_._2.toString) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-read-lz4-compressed-data-using-fileStream-of-spark-streaming-tp22868.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
