I didn't see the code snippet. Were you using picture(s) ? Please pastebin the code.
It would be better if you pastebin executor log for the killed executor. Thanks On Wed, May 18, 2016 at 9:41 PM, gkumar7 <gkum...@hawk.iit.edu> wrote: > I would like to test the latency (tasks/s) perceived in a simple > application > on Apache Spark. > > The idea: The workers will generate random data to be placed in a list. The > final action (count) will count the total number of data points generated. > > Below, the numberOfPartitions is equal to the number of datapoints which > need to be generated (datapoints are integers). > > Although the code works as expected, a total of 119 spark executors were > killed while running with 64 slaves. I feel this is because since spark > assigns executors to each node, the amount of total partitions each node is > assigned to compute may be larger than the available memory on that node. > This causes these executors to be killed and therefore, the latency > measurement I would like to analyze is inaccurate. > > Any assistance with code cleanup below or how to fix the above issue to > decrease the number of killed executors, would be much appreciated. > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Latency-experiment-without-losing-executors-tp26981.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
