Hi TD: The input file is on hdfs.
The file is approx 2.7 GB and when the process starts, there are 11 tasks (since hdfs block size is 256M) for processing and 2 tasks for reduce by key. After the file has been processed, I see new stages with 2 tasks that continue to be generated. I understand this value (2) is the default value for spark.default.parallelism but don't quite understand how is the value determined for generating tasks for reduceByKey, how is it used besides reduceByKey and what should be the optimal value for this. Thanks. On Thursday, July 10, 2014 7:24 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: How are you supplying the text file? On Wed, Jul 9, 2014 at 11:51 AM, M Singh <mans6si...@yahoo.com> wrote: Hi Folks: > > > >I am working on an application which uses spark streaming (version 1.1.0 >snapshot on a standalone cluster) to process text file and save counters in >cassandra based on fields in each row. I am testing the application in two >modes: > > * Process each row and save the counter in cassandra. In this scenario > after the text file has been consumed, there is no task/stages seen in the > spark UI. > > * If instead I use reduce by key before saving to cassandra, the spark > UI shows continuous generation of tasks/stages even after processing the file > has been completed. > >I believe this is because the reduce by key requires merging of data from >different partitions. But I was wondering if anyone has any insights/pointers >for understanding this difference in behavior and how to avoid generating >tasks/stages when there is no data (new file) available. > > >Thanks > >Mans
