Have you tried to increase the dirver memory?
On Thu, Jul 31, 2014 at 3:54 PM, Bin <wubin_phi...@126.com> wrote: > Hi All, > > The data size of my task is about 30mb. It runs smoothly in local mode. > However, when I submit it to the cluster, it throws the titled error > (Please see below for the complete output). > > Actually, my output is almost the same with > http://stackoverflow.com/questions/24080891/spark-program-hangs-at-job-finished-toarray-workers-throw-java-util-concurren. > I > also toArray my data, which was the reason of his case. > > However, how come it runs OK in local but not in the cluster? The memory > of each worker is over 60g, and my run command is: > > "$SPARK_HOME/bin/spark-class org.apache.spark.deploy.Client launch spark:// > 10.196.135.101:7077 $jar_path $programname -Dspark.master=spark:// > 10.196.135.101:7077 -Dspark.cores.max=300 -Dspark.executor.memory=20g > -spark.jars=$jar_path -Dspark.default.parallelism=100 > -Dspark.hadoop.hadoop.job.ugi=$username,$groupname -Dspark.app.name=$appname > $in_path $scala_out_path" > > Looking for help and thanks a lot! > > Below please find the complete output: > > 14/07/31 15:06:53 WARN Configuration: DEPRECATED: hadoop-site.xml found in > the classpath. Usage of hadoop-site.xml is deprecated. Instead use > core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of > core-default.xml, mapred-default.xml and hdfs-default.xml respectively > 14/07/31 15:06:53 INFO SecurityManager: Changing view acls to: spark > 14/07/31 15:06:53 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(spark) > 14/07/31 15:06:53 INFO Slf4jLogger: Slf4jLogger started > 14/07/31 15:06:53 INFO Remoting: Starting remoting > 14/07/31 15:06:54 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkExecutor@tdw-10-215-140-22:39446] > 14/07/31 15:06:54 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://sparkExecutor@tdw-10-215-140-22:39446] > 14/07/31 15:06:54 INFO CoarseGrainedExecutorBackend: Connecting to driver: > akka.tcp://spark@tdw-10-196-135-106:38502/user/CoarseGrainedScheduler > 14/07/31 15:06:54 INFO WorkerWatcher: Connecting to worker > akka.tcp://sparkWorker@tdw-10-215-140-22:34755/user/Worker > 14/07/31 15:06:54 INFO WorkerWatcher: Successfully connected to > akka.tcp://sparkWorker@tdw-10-215-140-22:34755/user/Worker > 14/07/31 15:06:56 INFO CoarseGrainedExecutorBackend: Successfully registered > with driver > 14/07/31 15:06:56 INFO SecurityManager: Changing view acls to: spark > 14/07/31 15:06:56 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(spark) > 14/07/31 15:06:56 INFO Slf4jLogger: Slf4jLogger started > 14/07/31 15:06:56 INFO Remoting: Starting remoting > 14/07/31 15:06:56 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://spark@tdw-10-215-140-22:56708] > 14/07/31 15:06:56 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://spark@tdw-10-215-140-22:56708] > 14/07/31 15:06:56 INFO SparkEnv: Connecting to MapOutputTracker: > akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker > 14/07/31 15:06:58 INFO SparkEnv: Connecting to BlockManagerMaster: > akka.tcp://spark@tdw-10-196-135-106:38502/user/BlockManagerMaster > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data1/sparkenv/local/spark-local-20140731150659-3f12 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data2/sparkenv/local/spark-local-20140731150659-1602 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data3/sparkenv/local/spark-local-20140731150659-d213 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data4/sparkenv/local/spark-local-20140731150659-f42e > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data5/sparkenv/local/spark-local-20140731150659-63d0 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data6/sparkenv/local/spark-local-20140731150659-9003 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data7/sparkenv/local/spark-local-20140731150659-f260 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data8/sparkenv/local/spark-local-20140731150659-6334 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data9/sparkenv/local/spark-local-20140731150659-3af4 > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data10/sparkenv/local/spark-local-20140731150659-133d > 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at > /data11/sparkenv/local/spark-local-20140731150659-ed08 > 14/07/31 15:06:59 INFO MemoryStore: MemoryStore started with capacity 11.5 GB. > 14/07/31 15:06:59 INFO ConnectionManager: Bound socket to port 35127 with id > = ConnectionManagerId(tdw-10-215-140-22,35127) > 14/07/31 15:06:59 INFO BlockManagerMaster: Trying to register BlockManager > 14/07/31 15:07:00 INFO BlockManagerMaster: Registered BlockManager > 14/07/31 15:07:00 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-0914d215-dd22-4d5e-9ec0-724937dbfd8b > 14/07/31 15:07:00 INFO HttpServer: Starting HTTP Server > 14/07/31 15:07:26 INFO CoarseGrainedExecutorBackend: Got assigned task 12 > 14/07/31 15:07:26 INFO CoarseGrainedExecutorBackend: Got assigned task 25 > 14/07/31 15:07:26 INFO Executor: Running task ID 25 > 14/07/31 15:07:26 INFO Executor: Running task ID 12 > 14/07/31 15:07:26 INFO Executor: Fetching > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/adsorption_2.10-1.0.jar > with timestamp 1406790410442 > 14/07/31 15:07:26 INFO Executor: Adding > file:/data/home/spark/spark-1.0.0-bin-0.20.2-cdh3u3/work/app-20140731150652-4911/8/./adsorption_2.10-1.0.jar > to class loader > 14/07/31 15:07:26 INFO HttpBroadcast: Started reading broadcast variable 0 > 14/07/31 15:07:28 INFO MemoryStore: ensureFreeSpace(102650) called with > curMem=0, maxMem=12348240691 > 14/07/31 15:07:28 INFO MemoryStore: Block broadcast_0 stored as values to > memory (estimated size 100.2 KB, free 11.5 GB) > 14/07/31 15:07:28 INFO HttpBroadcast: Reading broadcast variable 0 took > 1.702291406 s > 14/07/31 15:07:28 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:28 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00072:0+197955 > 14/07/31 15:07:28 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00062:0+218630 > 14/07/31 15:07:28 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 14/07/31 15:07:28 WARN LoadSnappy: Snappy native library not loaded > 14/07/31 15:07:29 INFO Executor: Serialized size of result for 12 is 887 > 14/07/31 15:07:29 INFO Executor: Serialized size of result for 25 is 887 > 14/07/31 15:07:29 INFO Executor: Sending result for 25 directly to driver > 14/07/31 15:07:29 INFO Executor: Sending result for 12 directly to driver > 14/07/31 15:07:29 INFO Executor: Finished task ID 12 > 14/07/31 15:07:29 INFO Executor: Finished task ID 25 > 14/07/31 15:07:30 INFO CoarseGrainedExecutorBackend: Got assigned task 30 > 14/07/31 15:07:30 INFO Executor: Running task ID 30 > 14/07/31 15:07:30 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:30 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00084:0+196433 > 14/07/31 15:07:30 INFO Executor: Serialized size of result for 30 is 887 > 14/07/31 15:07:30 INFO Executor: Sending result for 30 directly to driver > 14/07/31 15:07:30 INFO Executor: Finished task ID 30 > 14/07/31 15:07:30 INFO CoarseGrainedExecutorBackend: Got assigned task 31 > 14/07/31 15:07:30 INFO Executor: Running task ID 31 > 14/07/31 15:07:30 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:30 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00089:0+190194 > 14/07/31 15:07:30 INFO Executor: Serialized size of result for 31 is 887 > 14/07/31 15:07:30 INFO Executor: Sending result for 31 directly to driver > 14/07/31 15:07:30 INFO Executor: Finished task ID 31 > 14/07/31 15:07:31 INFO CoarseGrainedExecutorBackend: Got assigned task 54 > 14/07/31 15:07:31 INFO Executor: Running task ID 54 > 14/07/31 15:07:31 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:31 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00096:0+153443 > 14/07/31 15:07:31 INFO CoarseGrainedExecutorBackend: Got assigned task 55 > 14/07/31 15:07:31 INFO Executor: Running task ID 55 > 14/07/31 15:07:31 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:31 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00139:0+174726 > 14/07/31 15:07:31 INFO Executor: Serialized size of result for 54 is 887 > 14/07/31 15:07:31 INFO Executor: Sending result for 54 directly to driver > 14/07/31 15:07:31 INFO Executor: Finished task ID 54 > 14/07/31 15:07:31 INFO Executor: Serialized size of result for 55 is 887 > 14/07/31 15:07:31 INFO Executor: Sending result for 55 directly to driver > 14/07/31 15:07:31 INFO Executor: Finished task ID 55 > 14/07/31 15:07:32 INFO CoarseGrainedExecutorBackend: Got assigned task 76 > 14/07/31 15:07:32 INFO Executor: Running task ID 76 > 14/07/31 15:07:32 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:32 INFO CoarseGrainedExecutorBackend: Got assigned task 79 > 14/07/31 15:07:32 INFO Executor: Running task ID 79 > 14/07/31 15:07:32 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:32 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00149:0+134758 > 14/07/31 15:07:32 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00157:0+103176 > 14/07/31 15:07:32 INFO Executor: Serialized size of result for 79 is 887 > 14/07/31 15:07:32 INFO Executor: Sending result for 79 directly to driver > 14/07/31 15:07:32 INFO Executor: Finished task ID 79 > 14/07/31 15:07:32 INFO Executor: Serialized size of result for 76 is 887 > 14/07/31 15:07:32 INFO Executor: Sending result for 76 directly to driver > 14/07/31 15:07:32 INFO Executor: Finished task ID 76 > 14/07/31 15:07:34 INFO CoarseGrainedExecutorBackend: Got assigned task 99 > 14/07/31 15:07:34 INFO Executor: Running task ID 99 > 14/07/31 15:07:34 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:34 INFO HadoopRDD: Input split: > hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00167:0+51005 > 14/07/31 15:07:34 INFO Executor: Serialized size of result for 99 is 887 > 14/07/31 15:07:34 INFO Executor: Sending result for 99 directly to driver > 14/07/31 15:07:34 INFO Executor: Finished task ID 99 > 14/07/31 15:07:39 INFO CoarseGrainedExecutorBackend: Got assigned task 181 > 14/07/31 15:07:39 INFO Executor: Running task ID 181 > 14/07/31 15:07:39 INFO CoarseGrainedExecutorBackend: Got assigned task 196 > 14/07/31 15:07:39 INFO Executor: Running task ID 196 > 14/07/31 15:07:39 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:39 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Updating epoch to 1 and > clearing cache > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-24/10.215.140.24] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-196-135-105/10.196.135.105] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-21/10.215.140.21] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-12/10.215.140.12] > 46.947: [GC 10486272K->32824K(40196096K), 0.0347340 secs] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-13/10.215.140.13] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-23/10.215.140.23] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-13/10.215.140.13:58657] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-23/10.215.140.23:39188] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-21/10.215.140.21:36128] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-12/10.215.140.12:33380] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-196-135-105/10.196.135.105:36859] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-24/10.215.140.24:49100] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-23/10.215.140.23:39188], 2 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-13/10.215.140.13:58657], 2 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-196-135-105/10.196.135.105:36859], 2 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-12/10.215.140.12:33380], 1 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-21/10.215.140.21:36128], 1 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-24/10.215.140.24:49100], 2 messages pending > 14/07/31 15:07:39 INFO CacheManager: Partition rdd_6_10 not found, computing > it > 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Don't have map outputs for > shuffle 0, fetching them > 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor > = > Actor[akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker#-128956169] > 14/07/31 15:07:39 INFO CacheManager: Partition rdd_6_25 not found, computing > it > 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Don't have map outputs for > shuffle 0, fetching them > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-17/10.215.140.17] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-25/10.215.140.25] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-17/10.215.140.17:35040] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-17/10.215.140.17:35040], 1 messages pending > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-18/10.215.140.18] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-25/10.215.140.25:50298] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-25/10.215.140.25:50298], 2 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-18/10.215.140.18:33575] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-18/10.215.140.18:33575], 1 messages pending > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-14/10.215.140.14] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-14/10.215.140.14:37220] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-14/10.215.140.14:37220], 1 messages pending > 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Got the output locations > 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-196-135-107/10.196.135.107:59290] > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-11/10.215.140.11:50302] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-11/10.215.140.11:50302], 1 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-196-135-106/10.196.135.106:34128] > 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-11/10.215.140.11] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-196-135-107/10.196.135.107:59290], 2 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-196-135-106/10.196.135.106:34128], 1 messages pending > 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-15/10.215.140.15:50069] > 14/07/31 15:07:39 INFO SendingConnection: Connected to > [tdw-10-215-140-15/10.215.140.15:50069], 1 messages pending > 14/07/31 15:07:40 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 48 ms > 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-15/10.215.140.15] > 14/07/31 15:07:40 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 49 ms > 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from > [tdw-10-215-140-16/10.215.140.16] > 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from > [tdw-10-196-135-102/10.196.135.102] > 14/07/31 15:07:40 INFO SendingConnection: Initiating connection to > [tdw-10-215-140-16/10.215.140.16:58648] > 14/07/31 15:07:40 INFO SendingConnection: Connected to > [tdw-10-215-140-16/10.215.140.16:58648], 1 messages pending > 14/07/31 15:07:40 INFO SendingConnection: Initiating connection to > [tdw-10-196-135-102/10.196.135.102:45729] > 14/07/31 15:07:40 INFO SendingConnection: Connected to > [tdw-10-196-135-102/10.196.135.102:45729], 1 messages pending > 14/07/31 15:07:42 INFO ConnectionManager: Accepted connection from > [tdw-10-196-135-106/10.196.135.106] > 14/07/31 15:07:44 INFO ConnectionManager: Accepted connection from > [tdw-10-196-135-107/10.196.135.107] > 14/07/31 15:07:45 INFO MemoryStore: ensureFreeSpace(1922882) called with > curMem=102650, maxMem=12348240691 > 14/07/31 15:07:45 INFO MemoryStore: Block rdd_6_25 stored as values to memory > (estimated size 1877.8 KB, free 11.5 GB) > 14/07/31 15:07:46 INFO MemoryStore: ensureFreeSpace(1912396) called with > curMem=2025532, maxMem=12348240691 > 14/07/31 15:07:46 INFO MemoryStore: Block rdd_6_10 stored as values to memory > (estimated size 1867.6 KB, free 11.5 GB) > 14/07/31 15:07:46 INFO BlockManagerMaster: Updated info of block rdd_6_10 > 14/07/31 15:07:46 INFO BlockManagerMaster: Updated info of block rdd_6_25 > 14/07/31 15:07:46 INFO Executor: Serialized size of result for 181 is 421363 > 14/07/31 15:07:46 INFO Executor: Serialized size of result for 196 is 421522 > 14/07/31 15:07:46 INFO Executor: Sending result for 181 directly to driver > 14/07/31 15:07:46 INFO Executor: Sending result for 196 directly to driver > 14/07/31 15:07:46 INFO Executor: Finished task ID 181 > 14/07/31 15:07:46 INFO Executor: Finished task ID 196 > 14/07/31 15:07:50 INFO CoarseGrainedExecutorBackend: Got assigned task 219 > 14/07/31 15:07:50 INFO Executor: Running task ID 219 > 14/07/31 15:07:50 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:50 INFO CacheManager: Partition rdd_6_48 not found, computing > it > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 19 ms > 14/07/31 15:07:50 INFO CoarseGrainedExecutorBackend: Got assigned task 225 > 14/07/31 15:07:50 INFO Executor: Running task ID 225 > 14/07/31 15:07:50 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:50 INFO CacheManager: Partition rdd_6_54 not found, computing > it > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 15 ms > 14/07/31 15:07:51 INFO MemoryStore: ensureFreeSpace(1927469) called with > curMem=3937928, maxMem=12348240691 > 14/07/31 15:07:51 INFO MemoryStore: Block rdd_6_48 stored as values to memory > (estimated size 1882.3 KB, free 11.5 GB) > 14/07/31 15:07:51 INFO BlockManagerMaster: Updated info of block rdd_6_48 > 14/07/31 15:07:51 INFO Executor: Serialized size of result for 219 is 424342 > 14/07/31 15:07:51 INFO Executor: Sending result for 219 directly to driver > 14/07/31 15:07:51 INFO Executor: Finished task ID 219 > 14/07/31 15:07:51 INFO MemoryStore: ensureFreeSpace(1909775) called with > curMem=5865397, maxMem=12348240691 > 14/07/31 15:07:51 INFO MemoryStore: Block rdd_6_54 stored as values to memory > (estimated size 1865.0 KB, free 11.5 GB) > 14/07/31 15:07:51 INFO BlockManagerMaster: Updated info of block rdd_6_54 > 14/07/31 15:07:51 INFO Executor: Serialized size of result for 225 is 421546 > 14/07/31 15:07:51 INFO Executor: Sending result for 225 directly to driver > 14/07/31 15:07:51 INFO Executor: Finished task ID 225 > 14/07/31 15:07:53 INFO CoarseGrainedExecutorBackend: Got assigned task 251 > 14/07/31 15:07:53 INFO Executor: Running task ID 251 > 14/07/31 15:07:53 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:53 INFO CacheManager: Partition rdd_6_80 not found, computing > it > 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 15 ms > 14/07/31 15:07:53 INFO MemoryStore: ensureFreeSpace(1927469) called with > curMem=7775172, maxMem=12348240691 > 14/07/31 15:07:53 INFO MemoryStore: Block rdd_6_80 stored as values to memory > (estimated size 1882.3 KB, free 11.5 GB) > 14/07/31 15:07:53 INFO BlockManagerMaster: Updated info of block rdd_6_80 > 14/07/31 15:07:53 INFO Executor: Serialized size of result for 251 is 424634 > 14/07/31 15:07:53 INFO Executor: Sending result for 251 directly to driver > 14/07/31 15:07:53 INFO Executor: Finished task ID 251 > 14/07/31 15:07:54 INFO CoarseGrainedExecutorBackend: Got assigned task 259 > 14/07/31 15:07:54 INFO Executor: Running task ID 259 > 14/07/31 15:07:54 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:54 INFO CacheManager: Partition rdd_6_88 not found, computing > it > 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 171 non-empty blocks out of 171 blocks > 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 14 remote fetches in 13 ms > 14/07/31 15:07:54 INFO MemoryStore: ensureFreeSpace(1921571) called with > curMem=9702641, maxMem=12348240691 > 14/07/31 15:07:54 INFO MemoryStore: Block rdd_6_88 stored as values to memory > (estimated size 1876.5 KB, free 11.5 GB) > 14/07/31 15:07:54 INFO BlockManagerMaster: Updated info of block rdd_6_88 > 14/07/31 15:07:54 INFO Executor: Serialized size of result for 259 is 418167 > 14/07/31 15:07:54 INFO Executor: Sending result for 259 directly to driver > 14/07/31 15:07:54 INFO Executor: Finished task ID 259 > 14/07/31 15:07:56 INFO CoarseGrainedExecutorBackend: Got assigned task 273 > 14/07/31 15:07:56 INFO Executor: Running task ID 273 > 14/07/31 15:07:56 INFO CoarseGrainedExecutorBackend: Got assigned task 290 > 14/07/31 15:07:56 INFO Executor: Running task ID 290 > 14/07/31 15:07:56 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:56 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:56 INFO BlockManager: Found block rdd_6_10 locally > 14/07/31 15:07:56 INFO BlockManager: Found block rdd_6_25 locally > 14/07/31 15:07:56 INFO Executor: Serialized size of result for 273 is 887 > 14/07/31 15:07:56 INFO Executor: Sending result for 273 directly to driver > 14/07/31 15:07:56 INFO Executor: Finished task ID 273 > 14/07/31 15:07:56 INFO Executor: Serialized size of result for 290 is 887 > 14/07/31 15:07:56 INFO Executor: Sending result for 290 directly to driver > 14/07/31 15:07:56 INFO Executor: Finished task ID 290 > 14/07/31 15:07:57 INFO CoarseGrainedExecutorBackend: Got assigned task 308 > 14/07/31 15:07:57 INFO Executor: Running task ID 308 > 14/07/31 15:07:57 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:57 INFO BlockManager: Found block rdd_6_48 locally > 14/07/31 15:07:57 INFO CoarseGrainedExecutorBackend: Got assigned task 311 > 14/07/31 15:07:57 INFO Executor: Running task ID 311 > 14/07/31 15:07:57 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:57 INFO BlockManager: Found block rdd_6_54 locally > 14/07/31 15:07:57 INFO Executor: Serialized size of result for 308 is 887 > 14/07/31 15:07:57 INFO Executor: Sending result for 308 directly to driver > 14/07/31 15:07:57 INFO Executor: Finished task ID 308 > 14/07/31 15:07:57 INFO Executor: Serialized size of result for 311 is 887 > 14/07/31 15:07:57 INFO Executor: Sending result for 311 directly to driver > 14/07/31 15:07:57 INFO Executor: Finished task ID 311 > 14/07/31 15:07:58 INFO CoarseGrainedExecutorBackend: Got assigned task 339 > 14/07/31 15:07:58 INFO Executor: Running task ID 339 > 14/07/31 15:07:58 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:58 INFO BlockManager: Found block rdd_6_80 locally > 14/07/31 15:07:58 INFO CoarseGrainedExecutorBackend: Got assigned task 341 > 14/07/31 15:07:58 INFO Executor: Running task ID 341 > 14/07/31 15:07:58 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:58 INFO BlockManager: Found block rdd_6_88 locally > 14/07/31 15:07:58 INFO Executor: Serialized size of result for 339 is 887 > 14/07/31 15:07:58 INFO Executor: Sending result for 339 directly to driver > 14/07/31 15:07:58 INFO Executor: Finished task ID 339 > 14/07/31 15:07:58 INFO Executor: Serialized size of result for 341 is 887 > 14/07/31 15:07:58 INFO Executor: Sending result for 341 directly to driver > 14/07/31 15:07:58 INFO Executor: Finished task ID 341 > 14/07/31 15:07:59 INFO CoarseGrainedExecutorBackend: Got assigned task 377 > 14/07/31 15:07:59 INFO Executor: Running task ID 377 > 14/07/31 15:07:59 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Updating epoch to 2 and > clearing cache > 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Don't have map outputs for > shuffle 1, fetching them > 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor > = > Actor[akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker#-128956169] > 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Got the output locations > 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 9 ms > 14/07/31 15:08:00 INFO CoarseGrainedExecutorBackend: Got assigned task 393 > 14/07/31 15:08:00 INFO Executor: Running task ID 393 > 14/07/31 15:08:00 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 8 ms > 14/07/31 15:08:00 INFO Executor: Serialized size of result for 377 is 303256 > 14/07/31 15:08:00 INFO Executor: Sending result for 377 directly to driver > 14/07/31 15:08:00 INFO Executor: Finished task ID 377 > 14/07/31 15:08:00 INFO Executor: Serialized size of result for 393 is 310660 > 14/07/31 15:08:00 INFO Executor: Sending result for 393 directly to driver > 14/07/31 15:08:00 INFO Executor: Finished task ID 393 > 14/07/31 15:08:01 INFO CoarseGrainedExecutorBackend: Got assigned task 403 > 14/07/31 15:08:01 INFO Executor: Running task ID 403 > 14/07/31 15:08:01 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 7 ms > 14/07/31 15:08:01 INFO Executor: Serialized size of result for 403 is 299667 > 14/07/31 15:08:01 INFO Executor: Sending result for 403 directly to driver > 14/07/31 15:08:01 INFO Executor: Finished task ID 403 > 14/07/31 15:08:02 INFO CoarseGrainedExecutorBackend: Got assigned task 412 > 14/07/31 15:08:02 INFO Executor: Running task ID 412 > 14/07/31 15:08:02 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 6 ms > 14/07/31 15:08:02 INFO Executor: Serialized size of result for 412 is 301593 > 14/07/31 15:08:02 INFO Executor: Sending result for 412 directly to driver > 14/07/31 15:08:02 INFO Executor: Finished task ID 412 > 14/07/31 15:08:04 INFO CoarseGrainedExecutorBackend: Got assigned task 437 > 14/07/31 15:08:04 INFO Executor: Running task ID 437 > 14/07/31 15:08:04 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 6 ms > 14/07/31 15:08:04 INFO Executor: Serialized size of result for 437 is 312543 > 14/07/31 15:08:04 INFO Executor: Sending result for 437 directly to driver > 14/07/31 15:08:04 INFO Executor: Finished task ID 437 > 14/07/31 15:08:04 INFO CoarseGrainedExecutorBackend: Got assigned task 445 > 14/07/31 15:08:04 INFO Executor: Running task ID 445 > 14/07/31 15:08:04 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 6 ms > 14/07/31 15:08:04 INFO Executor: Serialized size of result for 445 is 307049 > 14/07/31 15:08:04 INFO Executor: Sending result for 445 directly to driver > 14/07/31 15:08:04 INFO Executor: Finished task ID 445 > 14/07/31 15:08:06 INFO CoarseGrainedExecutorBackend: Got assigned task 467 > 14/07/31 15:08:06 INFO Executor: Running task ID 467 > 14/07/31 15:08:06 INFO BlockManager: Found block broadcast_0 locally > 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > maxBytesInFlight: 50331648, targetRequestSize: 10066329 > 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Getting 100 non-empty blocks out of 100 blocks > 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: > Started 16 remote fetches in 6 ms > 14/07/31 15:08:07 INFO Executor: Serialized size of result for 467 is 301177 > 14/07/31 15:08:07 INFO Executor: Sending result for 467 directly to driver > 14/07/31 15:08:07 INFO Executor: Finished task ID 467 > 14/07/31 15:08:18 INFO ShuffleBlockManager: Deleted all files for shuffle 1 > 14/07/31 15:09:00 WARN BlockManagerMaster: Error sending message to > BlockManagerMaster in 1 attempts*java.util.concurrent.TimeoutException: > Futures timed out after [30 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:237) > at > org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:51) > at org.apache.spark.storage.BlockManager.org > <http://org.apache.spark.storage.BlockManager.org>$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:113) > at > org.apache.spark.storage.BlockManager$$anonfun$initialize$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BlockManager.scala:158) > at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:790) > at > org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:158) > at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80) > at > akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744)* > 14/07/31 15:09:12 INFO ShuffleBlockManager: Deleted all files for shuffle 0 > 14/07/31 15:09:12 INFO BlockManager: Removing RDD 6 > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_88 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_88 of size 1921571 dropped > from memory (free 12338538050) > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_25 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_25 of size 1922882 dropped > from memory (free 12340460932) > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_10 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_10 of size 1912396 dropped > from memory (free 12342373328) > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_54 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_54 of size 1909775 dropped > from memory (free 12344283103) > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_80 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_80 of size 1927469 dropped > from memory (free 12346210572) > 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_48 > 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_48 of size 1927469 dropped > from memory (free 12348138041) > > > >
