If you look at the api doc <https://spark.apache.org/docs/1.1.0/api/scala/index.html#org.apache.spark.streaming.StreamingContext>, you can see the fileStream has a boolean parameter( newFilesOnly), setting it false would pick up the existing files it seems.
Thanks Best Regards On Sun, Nov 30, 2014 at 4:46 AM, yu <[email protected]> wrote: > Hello Everyone, > > I am learning spark streaming and hope to find a convenient way to generate > data stream from textfiles for some simple experiments. After I've viewed > the scaladoc of spark, I found the methods 'textFileStream' and > 'fileStream' > could only monitor new files coming in but not existing files. Is there any > method I could directly use in spark? For example, I have text1 in a > folder, > how can I generate DStream containing the data from text1? > > Thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Generating-a-DStream-by-existing-textfiles-tp20030.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
