Suraj I think your issue is the code is not detecting new files in the directory you set, a bit different from what I'm facing. If it detects a new file, you should see something similar in the logs:
14/02/18 19:17:30 INFO JobScheduler: Added jobs for time 1392751050000 ms 14/02/18 19:17:30 INFO JobScheduler: Starting job streaming job 1392751050000 ms.0 from job set of time 1392751050000 ms 14/02/18 19:17:30 WARN Configuration: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 14/02/18 19:17:30 WARN Configuration: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 14/02/18 19:17:30 WARN Configuration: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type 14/02/18 19:17:30 INFO SparkContext: Starting job: saveAsTextFile at web_reqeusts.scala:81 14/02/18 19:17:30 INFO SparkContext: Job finished: saveAsTextFile at web_reqeusts.scala:81, took 2.4977E-5 s 14/02/18 19:17:30 INFO JobScheduler: Finished job streaming job 1392751050000 ms.0 from job set of time 1392751050000 ms 14/02/18 19:17:30 INFO JobScheduler: Total delay: 0.036 s for time 1392751050000 ms (execution: 0.023 s) 14/02/18 19:17:30 INFO FileInputDStream: Cleared 1 old files that were older than 1392751020000 ms: 1392750990000 ms 14/02/18 19:18:00 INFO FileInputDStream: Finding new files took 8 ms 14/02/18 19:18:00 INFO FileInputDStream: New files at time 1392751080000 ms: *hdfs://nn1:8020/user/etl/rtp_sink/staging/0133-8f0e55d43a6c43bba48a97d4c448762a.sdb.gz hdfs://nn1:8020/user/etl/rtp_sink/staging/0133-ad3925c1ce04450abde4208b731aae1d.sdb.gz hdfs://nn1:8020/user/etl/rtp_sink/staging/0133-b9c49509293b459cae0715d5905e6805.sdb.gz* 14/02/18 19:18:00 INFO MemoryStore: ensureFreeSpace(170493) called with curMem=6308257, maxMem=9003781324 14/02/18 19:18:00 INFO MemoryStore: Block broadcast_37 stored as values to memory (estimated size 166.5 KB, free 8.4 GB) 14/02/18 19:18:00 INFO MemoryStore: ensureFreeSpace(170493) called with curMem=6478750, maxMem=9003781324 14/02/18 19:18:00 INFO MemoryStore: Block broadcast_38 stored as values to memory (estimated size 166.5 KB, free 8.4 GB) 14/02/18 19:18:00 INFO MemoryStore: ensureFreeSpace(170493) called with curMem=6649243, maxMem=9003781324 14/02/18 19:18:00 INFO MemoryStore: Block broadcast_39 stored as values to memory (estimated size 166.5 KB, free 8.4 GB) 14/02/18 19:18:00 INFO FileInputFormat: Total input paths to process : 1 14/02/18 19:18:00 INFO FileInputFormat: Total input paths to process : 1 14/02/18 19:18:00 INFO FileInputFormat: Total input paths to process : 1 ----- -- Robin Li -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DStream-saveAsTextFiles-saves-nothing-tp1666p1690.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
