I am not sure why you are getting node_local and not process_local. Also there is probably not a good documentation other than that configuration page - http://spark.apache.org/docs/latest/configuration.html (search for locality)
On Thu, Jul 9, 2015 at 5:51 AM, Michel Hubert <mich...@vsnsystemen.nl> wrote: > > > > > > > StorageLevel.MEMORY_ONLY > > > > Streaming / Batch Processing Statistics / Processing Time: > > Executor-cores NOT specified: 5s > > Executor-cores 8: 400 ms > > > > Processing time has significantly reduced with executor-cores set to 8. > > But what is the general rule of thumb for a good excecutor-cores settings? > > > > But 257 records/sec still isn’t very, or not? > > Statistics over last 95 processed batches Receiver Statistics > > · *Receiver* > > · *Status* > > · *Location* > > · *Records in last batch* > > · *[2015/07/09 14:49:47]* > > · *Minimum rate* > > · *[records/sec]* > > · *Median rate* > > · *[records/sec]* > > · *Maximum rate* > > · *[records/sec]* > > · *Last Error* > > SocketReceiver-0 > > ACTIVE > > bfravicsvr81440-cld.opentsp.com > > 1352 > > 0 > > 257 > > 279 > > - > > > > > > > > “After that the somehow the system is not able to launch any process > local task” > > Where can I look for an answer why some tasks are NODE_LOCAL and others > are PROCESS_LOCAL? > > Where should I look for the reason the locality wait expires? > > > Details for Stage 147 > > · *Total task time across all tasks:* 0.4 s > > · *Shuffle write:* 704.0 B / 26 > > *Show additional metrics* > Summary Metrics for 26 Completed Tasks > > *Metric* > > *Min* > > *25th percentile* > > *Median* > > *75th percentile* > > *Max* > > Duration > > 9 ms > > 11 ms > > 12 ms > > 24 ms > > 30 ms > > GC Time > > 0 ms > > 0 ms > > 0 ms > > 0 ms > > 0 ms > > Shuffle Write Size / Records > > 27.0 B / 1 > > 27.0 B / 1 > > 27.0 B / 1 > > 27.0 B / 1 > > 28.0 B / 1 > Aggregated Metrics by Executor > > *Executor ID* > > *Address* > > *Task Time* > > *Total Tasks* > > *Failed Tasks* > > *Succeeded Tasks* > > *Shuffle Write Size / Records* > > 1 > > xxx81440xxx:38882 > > 0.8 s > > 25 > > 0 > > 25 > > 677.0 B / 25 > > 2 > > xxx81441xxx:46832 > > 45 ms > > 1 > > 0 > > 1 > > 27.0 B / 1 > Tasks > > *Index* > > *ID* > > *Attempt* > > *Status* > > *Locality Level* > > *Executor ID / Host* > > *Launch Time* > > *Duration* > > *GC Time* > > *Write Time* > > *Shuffle Write Size / Records* > > *Errors* > > 0 > > 1029 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 30 ms > > 27.0 B / 1 > > 2 > > 1031 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 29 ms > > 27.0 B / 1 > > 1 > > 1030 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 30 ms > > 27.0 B / 1 > > 5 > > 1034 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 25 ms > > 27.0 B / 1 > > 4 > > 1033 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 28 ms > > 27.0 B / 1 > > 3 > > 1032 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 27 ms > > 27.0 B / 1 > > 25 > > 1036 > > 0 > > SUCCESS > > PROCESS_LOCAL > > 2 / xxx81441xxx > > 2015/07/09 14:43:55 > > 23 ms > > 27.0 B / 1 > > 6 > > 1035 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 24 ms > > 27.0 B / 1 > > 7 > > 1037 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 10 ms > > 27.0 B / 1 > > 8 > > 1038 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 12 ms > > 27.0 B / 1 > > 9 > > 1039 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 11 ms > > 27.0 B / 1 > > 10 > > 1040 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 11 ms > > 27.0 B / 1 > > 11 > > 1041 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > > 2015/07/09 14:43:55 > > 11 ms > > 27.0 B / 1 > > 12 > > 1042 > > 0 > > SUCCESS > > NODE_LOCAL > > 1 / xxx81440xxx > ... > > [Message clipped]
