Hi, This sounds more like a task for complex event processing using Spark Streaming or Storm. There you can basically define time windows, such as 5 minutes, and do some analyses for the data within this time window. Can you give an example what do you do during processing?
Best regards, Le 19 août 2014 05:31, "salemi" <[email protected]> a écrit : > Hi, > > Mine data source stores the incoming data every 10 second to hdfs. The > naming convention save-<timestamp>.csv (see below) > > drwxr-xr-x ali supergroup 0 B 0 0 B > save-1408396065000.csv > drwxr-xr-x ali supergroup 0 B 0 0 B > save-1408396070000.csv > drwxr-xr-x ali supergroup 0 B 0 0 B > save-1408396075000.csv > drwxr-xr-x ali supergroup 0 B 0 0 B > save-1408396080000.csv > > I would like to periodically (every 5min) read the files and process them. > is there a good example out there how to implement this? How do I know what > part of the data I have already processed? > > Thanks, > Ali > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-Identifying-and-skipping-processed-data-in-hdfs-tp12347.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
