Hi, Mine data source stores the incoming data every 10 second to hdfs. The naming convention save-<timestamp>.csv (see below)
drwxr-xr-x ali supergroup 0 B 0 0 B save-1408396065000.csv drwxr-xr-x ali supergroup 0 B 0 0 B save-1408396070000.csv drwxr-xr-x ali supergroup 0 B 0 0 B save-1408396075000.csv drwxr-xr-x ali supergroup 0 B 0 0 B save-1408396080000.csv I would like to periodically (every 5min) read the files and process them. is there a good example out there how to implement this? How do I know what part of the data I have already processed? Thanks, Ali -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-Identifying-and-skipping-processed-data-in-hdfs-tp12347.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
