>> Hi, Spark support:
>>
>> I am working on a research project which uses Spark on Amazon EC2 as the
>> cluster foundation for distributed computation.
>>
>> The project basically consumes some image and text files and project these
>> files to different features and index them for later query.
>>
>> Currently the image and text data are stored in S3 and our application
>> distributed the projectors (convert the data to features) to worker nodes
>> via SparkContext map function and the result is persisted in memory for
>> later usage.
>>
>> For the whole workflow, all the projectors are run in parallel while the
>> indexing is done in sequential mode (i.e. indexing is not running on worker
>> node). I tried different number of worker nodes in the experiments and
>> notice that the performance increase (in terms of time-consumption) when we
>> go choose to run with number of worker nodes 2, 3, 4, .... but the
>> performance turned bad when 10 worker nodes are applied and worse on 12
>> (time-consuming going uphill on 10 and 12) and it turned good again for 14,
>> 16 and 18 worker nodes (time-consuming going downhill again).
>>
>> I tried another task (similar thing but different program) where the
>> intermediate features are used for training and finally classification is
>> applied for some test data against the trained model.
>> In this set of experiments we chose number of work nodes as 1, 2, 3, 4,
>> ...10. The same thing happened except the performance is more bumpy as the
>> time consumption went worse on 5 worker nodes than 4 and turned good on 6
>> worker nodes and turned bad on 8 worker nodes and turned back good on 9 and
>> 10 worker nodes.
>>
>> This is confusing as I am expecting the performance goes near-linear when we
>> increased the number of worker nodes. When the overhead is more than the
>> distributed gain, the performance goes bad and should not come back to good.
>>
>> P. S.
>>
>> I did make the below fine-tune steps:
>>
>> System.setProperty("spark.scheduler.mode", "FAIR") as I want each job
>> received more evenly assigned cluster resource
>> System.setProperty("spark.task.maxFailures", "6") as I want the worker node
>> tried more times before it fail on a task
>> System.setProperty("spark.akka.frameSize", "250") as our communication
>> between master and worker is more than the default 10M
>>
>> Did I miss any other tuning steps?
>>
>> Thanks,
>> Xiaobing
