Re: Initial tasks in job take time

2015-04-28 Thread Anshul Singhle
yes On 29 Apr 2015 03:31, "ayan guha" wrote: > Are your driver running on the same m/c as master? > On 29 Apr 2015 03:59, "Anshul Singhle" wrote: > >> Hi, >> >> I'm running short spark jobs on rdds cached in memory. I'm also using a >> long running job context. I want to be able to complete my j

Re: Initial tasks in job take time

2015-04-28 Thread ayan guha
Are your driver running on the same m/c as master? On 29 Apr 2015 03:59, "Anshul Singhle" wrote: > Hi, > > I'm running short spark jobs on rdds cached in memory. I'm also using a > long running job context. I want to be able to complete my jobs (on the > cached rdd) in under 1 sec. > I'm getting

Initial tasks in job take time

2015-04-28 Thread Anshul Singhle
Hi, I'm running short spark jobs on rdds cached in memory. I'm also using a long running job context. I want to be able to complete my jobs (on the cached rdd) in under 1 sec. I'm getting the following job times with about 15 GB of data distributed across 6 nodes. Each executor has about 20GB of m