Hi everyone,
I'm having problems with loading files. Either with java code or
spark-shell, I got same errors when I try to load a text file.
I added hadoop-client and hadoop-common 2.0.0-cdh4.6.0 as dependencies and
maven-shade-plugin is configured.
I have CDH 4.6.0, spark-0.9.1-bin-cdh4 and JDK7 installed on my nodes.
Here is my code:
public class CountTool {
public static void main(String[] args) {
// validate input
if (args.length != 1) {
System.err.println("invalid usage!\ncount <input>");
return;
}
// input parameters
String input = args[0];
// spark context
SparkConf conf = new SparkConf()
.setMaster("spark://server08.mydomain.com:7077")
.setSparkHome("/opt/spark")
.setJars(new String[] {"/home/hadoop/dist/mycode.jar"})
.setAppName(CountTool.class.getSimpleName())
.set("spark.executor.memory", "1g");
JavaSparkContext jsc = new JavaSparkContext(conf);
// fix for cdh4
jsc.hadoopConfiguration().set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
jsc.hadoopConfiguration().set("fs.file.impl",
org.apache.hadoop.fs.LocalFileSystem.class.getName());
// load file and count
JavaRDD<String> file = jsc.textFile(input);
long count = file.count();
System.out.println(String.format("file %s %d lines", input, count));
}
}
I couldn't load any file either from local disk or hdfs with this code. When
I try to load a file from local it gives "Loss was due to
java.io.EOFException" error:
[hadoop@server08 clustering]$ java -cp /home/hadoop/dist/mycode.jar
com.gg.clustering.spark.CountTool /home/hadoop/scripts/clustering/deals.txt
2014-05-20 10:04:22 INFO Slf4jLogger:80 - Slf4jLogger started
2014-05-20 10:04:22 INFO Remoting:74 - Starting remoting
2014-05-20 10:04:23 INFO Remoting:74 - Remoting started; listening on
addresses :[akka.tcp://sp...@server08.mydomain.com:34236]
2014-05-20 10:04:23 INFO Remoting:74 - Remoting now listens on addresses:
[akka.tcp://sp...@server08.mydomain.com:34236]
2014-05-20 10:04:23 INFO SparkEnv:50 - Registering BlockManagerMaster
2014-05-20 10:04:23 INFO DiskBlockManager:50 - Created local directory at
/tmp/spark-local-20140520100423-ae57
2014-05-20 10:04:23 INFO MemoryStore:50 - MemoryStore started with capacity
8.4 GB.
2014-05-20 10:04:23 INFO ConnectionManager:50 - Bound socket to port 58899
with id = ConnectionManagerId(server08.mydomain.com,58899)
2014-05-20 10:04:23 INFO BlockManagerMaster:50 - Trying to register
BlockManager
2014-05-20 10:04:23 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server08.mydomain.com:58899 with 8.4 GB RAM
2014-05-20 10:04:23 INFO BlockManagerMaster:50 - Registered BlockManager
2014-05-20 10:04:23 INFO HttpServer:50 - Starting HTTP Server
2014-05-20 10:04:23 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO AbstractConnector:338 - Started
SocketConnector@0.0.0.0:38679
2014-05-20 10:04:23 INFO HttpBroadcast:50 - Broadcast server started at
http://10.35.74.148:38679
2014-05-20 10:04:23 INFO SparkEnv:50 - Registering MapOutputTracker
2014-05-20 10:04:23 INFO HttpFileServer:50 - HTTP File server directory is
/tmp/spark-1e8e7419-9d16-446e-a7e2-5e0e6307f7d0
2014-05-20 10:04:23 INFO HttpServer:50 - Starting HTTP Server
2014-05-20 10:04:23 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO AbstractConnector:338 - Started
SocketConnector@0.0.0.0:42538
2014-05-20 10:04:23 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/stage,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/pool,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/environment,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/executors,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/metrics/json,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/static,null}
2014-05-20 10:04:23 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/,null}
2014-05-20 10:04:23 INFO AbstractConnector:338 - Started
SelectChannelConnector@0.0.0.0:4040
2014-05-20 10:04:23 INFO SparkUI:50 - Started Spark Web UI at
http://server08.mydomain.com:4040
2014-05-20 10:04:23 WARN NativeCodeLoader:62 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2014-05-20 10:04:24 INFO SparkContext:50 - Added JAR
/home/hadoop/dist/mycode.jar at http://10.35.74.148:42538/jars/mycode.jar
with timestamp 1400569464072
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Connecting to master
spark://server08.mydomain.com:7077...
2014-05-20 10:04:24 INFO MemoryStore:50 - ensureFreeSpace(124827) called
with curMem=0, maxMem=9006219264
2014-05-20 10:04:24 INFO MemoryStore:50 - Block broadcast_0 stored as
values to memory (estimated size 121.9 KB, free 8.4 GB)
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Connected to
Spark cluster with app ID app-20140520100424-0005
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/0 on
worker-20140520085014-server02.mydomain.com-54319
(server02.mydomain.com:54319) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/0 on hostPort server02.mydomain.com:54319 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/1 on
worker-20140520085014-server06.mydomain.com-33290
(server06.mydomain.com:33290) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/1 on hostPort server06.mydomain.com:33290 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/2 on
worker-20140520085014-server03.mydomain.com-50690
(server03.mydomain.com:50690) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/2 on hostPort server03.mydomain.com:50690 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/3 on
worker-20140520085014-server04.mydomain.com-41041
(server04.mydomain.com:41041) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/3 on hostPort server04.mydomain.com:41041 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/4 on
worker-20140520085014-server07.mydomain.com-44641
(server07.mydomain.com:44641) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/4 on hostPort server07.mydomain.com:44641 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/5 on
worker-20140520085014-server05.mydomain.com-32908
(server05.mydomain.com:32908) with 16 cores
2014-05-20 10:04:24 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/5 on hostPort server05.mydomain.com:32908 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/0 is now RUNNING
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/1 is now RUNNING
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/2 is now RUNNING
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/3 is now RUNNING
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/4 is now RUNNING
2014-05-20 10:04:24 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/5 is now RUNNING
2014-05-20 10:04:24 INFO FileInputFormat:233 - Total input paths to process
: 1
2014-05-20 10:04:24 INFO SparkContext:50 - Starting job: count at
CountTool.java:44
2014-05-20 10:04:24 INFO DAGScheduler:50 - Got job 0 (count at
CountTool.java:44) with 2 output partitions (allowLocal=false)
2014-05-20 10:04:24 INFO DAGScheduler:50 - Final stage: Stage 0 (count at
CountTool.java:44)
2014-05-20 10:04:24 INFO DAGScheduler:50 - Parents of final stage: List()
2014-05-20 10:04:24 INFO DAGScheduler:50 - Missing parents: List()
2014-05-20 10:04:24 INFO DAGScheduler:50 - Submitting Stage 0 (MappedRDD[1]
at textFile at CountTool.java:43), which has no missing parents
2014-05-20 10:04:24 INFO DAGScheduler:50 - Submitting 2 missing tasks from
Stage 0 (MappedRDD[1] at textFile at CountTool.java:43)
2014-05-20 10:04:24 INFO TaskSchedulerImpl:50 - Adding task set 0.0 with 2
tasks
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server03.mydomain.com:39494/user/Executor#1298593046]
with ID 2
2014-05-20 10:04:25 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 0
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:25 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 4 ms
2014-05-20 10:04:25 INFO TaskSetManager:50 - Starting task 0.0:1 as TID 1
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:25 INFO TaskSetManager:50 - Serialized task 0.0:1 as 1629
bytes in 0 ms
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server02.mydomain.com:59496/user/Executor#1432291773]
with ID 0
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server06.mydomain.com:39547/user/Executor#78156672]
with ID 1
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server05.mydomain.com:34135/user/Executor#-777886910]
with ID 5
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server07.mydomain.com:41161/user/Executor#336519762]
with ID 4
2014-05-20 10:04:25 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server04.mydomain.com:34174/user/Executor#-1382135714]
with ID 3
2014-05-20 10:04:25 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server03.mydomain.com:58285 with 589.2 MB RAM
2014-05-20 10:04:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server02.mydomain.com:51425 with 589.2 MB RAM
2014-05-20 10:04:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server06.mydomain.com:46383 with 589.2 MB RAM
2014-05-20 10:04:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server05.mydomain.com:52167 with 589.2 MB RAM
2014-05-20 10:04:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server04.mydomain.com:41308 with 589.2 MB RAM
2014-05-20 10:04:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server07.mydomain.com:42993 with 589.2 MB RAM
2014-05-20 10:04:27 WARN TaskSetManager:62 - Lost TID 1 (task 0.0:1)
2014-05-20 10:04:27 WARN TaskSetManager:62 - Loss was due to
java.io.EOFException
java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2744)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1032)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:145)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:62)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-20 10:04:27 WARN TaskSetManager:62 - Lost TID 0 (task 0.0:0)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 1]
2014-05-20 10:04:27 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 2
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 INFO TaskSetManager:50 - Starting task 0.0:1 as TID 3
on executor 5: server05.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Serialized task 0.0:1 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN TaskSetManager:62 - Lost TID 2 (task 0.0:0)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 2]
2014-05-20 10:04:27 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 4
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN TaskSetManager:62 - Lost TID 4 (task 0.0:0)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 3]
2014-05-20 10:04:27 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 5
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN TaskSetManager:62 - Lost TID 5 (task 0.0:0)
2014-05-20 10:04:27 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 4]
2014-05-20 10:04:27 ERROR TaskSetManager:66 - Task 0.0:0 failed 4 times;
aborting job
2014-05-20 10:04:27 INFO DAGScheduler:50 - Failed to run count at
CountTool.java:44
Exception in thread "main" org.apache.spark.SparkException: Job aborted:
Task 0.0:0 failed 4 times (most recent failure: Exception failure:
java.io.EOFException)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I got same error with hdfs:
[hadoop@server08 clustering]$ java -cp /home/hadoop/dist/mycode.jar
com.gg.clustering.spark.CountTool
hdfs://server01.mydomain.com:/user/hadoop/results/clustering/deals.txt
2014-05-20 10:05:26 INFO Slf4jLogger:80 - Slf4jLogger started
2014-05-20 10:05:26 INFO Remoting:74 - Starting remoting
2014-05-20 10:05:26 INFO Remoting:74 - Remoting started; listening on
addresses :[akka.tcp://sp...@server08.mydomain.com:55747]
2014-05-20 10:05:26 INFO Remoting:74 - Remoting now listens on addresses:
[akka.tcp://sp...@server08.mydomain.com:55747]
2014-05-20 10:05:26 INFO SparkEnv:50 - Registering BlockManagerMaster
2014-05-20 10:05:26 INFO DiskBlockManager:50 - Created local directory at
/tmp/spark-local-20140520100526-e8cb
2014-05-20 10:05:26 INFO MemoryStore:50 - MemoryStore started with capacity
8.4 GB.
2014-05-20 10:05:26 INFO ConnectionManager:50 - Bound socket to port 39356
with id = ConnectionManagerId(server08.mydomain.com,39356)
2014-05-20 10:05:26 INFO BlockManagerMaster:50 - Trying to register
BlockManager
2014-05-20 10:05:26 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server08.mydomain.com:39356 with 8.4 GB RAM
2014-05-20 10:05:26 INFO BlockManagerMaster:50 - Registered BlockManager
2014-05-20 10:05:26 INFO HttpServer:50 - Starting HTTP Server
2014-05-20 10:05:26 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO AbstractConnector:338 - Started
SocketConnector@0.0.0.0:53241
2014-05-20 10:05:26 INFO HttpBroadcast:50 - Broadcast server started at
http://10.35.74.148:53241
2014-05-20 10:05:26 INFO SparkEnv:50 - Registering MapOutputTracker
2014-05-20 10:05:26 INFO HttpFileServer:50 - HTTP File server directory is
/tmp/spark-ba35f7bd-9741-40e3-a814-37dbb9e2592e
2014-05-20 10:05:26 INFO HttpServer:50 - Starting HTTP Server
2014-05-20 10:05:26 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO AbstractConnector:338 - Started
SocketConnector@0.0.0.0:39988
2014-05-20 10:05:26 INFO Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/stage,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/pool,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/environment,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/executors,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/metrics/json,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/static,null}
2014-05-20 10:05:26 INFO ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/,null}
2014-05-20 10:05:26 INFO AbstractConnector:338 - Started
SelectChannelConnector@0.0.0.0:4040
2014-05-20 10:05:26 INFO SparkUI:50 - Started Spark Web UI at
http://server08.mydomain.com:4040
2014-05-20 10:05:26 WARN NativeCodeLoader:62 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2014-05-20 10:05:27 INFO SparkContext:50 - Added JAR
/home/hadoop/dist/mycode.jar at http://10.35.74.148:39988/jars/mycode.jar
with timestamp 1400569527177
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Connecting to master
spark://server08.mydomain.com:7077...
2014-05-20 10:05:27 INFO MemoryStore:50 - ensureFreeSpace(124827) called
with curMem=0, maxMem=9006219264
2014-05-20 10:05:27 INFO MemoryStore:50 - Block broadcast_0 stored as
values to memory (estimated size 121.9 KB, free 8.4 GB)
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Connected to
Spark cluster with app ID app-20140520100527-0006
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/0 on
worker-20140520085014-server02.mydomain.com-54319
(server02.mydomain.com:54319) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/0 on hostPort server02.mydomain.com:54319 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/1 on
worker-20140520085014-server06.mydomain.com-33290
(server06.mydomain.com:33290) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/1 on hostPort server06.mydomain.com:33290 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/2 on
worker-20140520085014-server03.mydomain.com-50690
(server03.mydomain.com:50690) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/2 on hostPort server03.mydomain.com:50690 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/3 on
worker-20140520085014-server04.mydomain.com-41041
(server04.mydomain.com:41041) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/3 on hostPort server04.mydomain.com:41041 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/4 on
worker-20140520085014-server07.mydomain.com-44641
(server07.mydomain.com:44641) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/4 on hostPort server07.mydomain.com:44641 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/5 on
worker-20140520085014-server05.mydomain.com-32908
(server05.mydomain.com:32908) with 16 cores
2014-05-20 10:05:27 INFO SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/5 on hostPort server05.mydomain.com:32908 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/1 is now RUNNING
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/0 is now RUNNING
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/5 is now RUNNING
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/2 is now RUNNING
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/3 is now RUNNING
2014-05-20 10:05:27 INFO AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/4 is now RUNNING
2014-05-20 10:05:28 INFO FileInputFormat:233 - Total input paths to process
: 1
2014-05-20 10:05:28 INFO SparkContext:50 - Starting job: count at
CountTool.java:44
2014-05-20 10:05:28 INFO DAGScheduler:50 - Got job 0 (count at
CountTool.java:44) with 2 output partitions (allowLocal=false)
2014-05-20 10:05:28 INFO DAGScheduler:50 - Final stage: Stage 0 (count at
CountTool.java:44)
2014-05-20 10:05:28 INFO DAGScheduler:50 - Parents of final stage: List()
2014-05-20 10:05:28 INFO DAGScheduler:50 - Missing parents: List()
2014-05-20 10:05:28 INFO DAGScheduler:50 - Submitting Stage 0 (MappedRDD[1]
at textFile at CountTool.java:43), which has no missing parents
2014-05-20 10:05:28 INFO DAGScheduler:50 - Submitting 2 missing tasks from
Stage 0 (MappedRDD[1] at textFile at CountTool.java:43)
2014-05-20 10:05:28 INFO TaskSchedulerImpl:50 - Adding task set 0.0 with 2
tasks
2014-05-20 10:05:28 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server03.mydomain.com:40916/user/Executor#1474912]
with ID 2
2014-05-20 10:05:28 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 0
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:05:28 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 5 ms
2014-05-20 10:05:28 INFO TaskSetManager:50 - Starting task 0.0:1 as TID 1
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:05:28 INFO TaskSetManager:50 - Serialized task 0.0:1 as 1692
bytes in 0 ms
2014-05-20 10:05:28 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server02.mydomain.com:42834/user/Executor#388749186]
with ID 0
2014-05-20 10:05:29 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server05.mydomain.com:49626/user/Executor#1815450843]
with ID 5
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server03.mydomain.com:49424 with 589.2 MB RAM
2014-05-20 10:05:29 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server07.mydomain.com:35109/user/Executor#-199036165]
with ID 4
2014-05-20 10:05:29 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server06.mydomain.com:35535/user/Executor#-1925679390]
with ID 1
2014-05-20 10:05:29 INFO SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkexecu...@server04.mydomain.com:53806/user/Executor#725612946]
with ID 3
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server02.mydomain.com:35927 with 589.2 MB RAM
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server05.mydomain.com:57374 with 589.2 MB RAM
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server07.mydomain.com:50669 with 589.2 MB RAM
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server06.mydomain.com:33799 with 589.2 MB RAM
2014-05-20 10:05:29 INFO BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server04.mydomain.com:53457 with 589.2 MB RAM
2014-05-20 10:05:30 WARN TaskSetManager:62 - Lost TID 1 (task 0.0:1)
2014-05-20 10:05:30 WARN TaskSetManager:62 - Loss was due to
java.io.EOFException
java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2744)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1032)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:145)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:62)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-20 10:05:30 WARN TaskSetManager:62 - Lost TID 0 (task 0.0:0)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 1]
2014-05-20 10:05:30 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 2
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 1 ms
2014-05-20 10:05:30 INFO TaskSetManager:50 - Starting task 0.0:1 as TID 3
on executor 5: server05.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Serialized task 0.0:1 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN TaskSetManager:62 - Lost TID 2 (task 0.0:0)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 2]
2014-05-20 10:05:30 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 4
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN TaskSetManager:62 - Lost TID 4 (task 0.0:0)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 3]
2014-05-20 10:05:30 INFO TaskSetManager:50 - Starting task 0.0:0 as TID 5
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN TaskSetManager:62 - Lost TID 5 (task 0.0:0)
2014-05-20 10:05:30 INFO TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 4]
2014-05-20 10:05:30 ERROR TaskSetManager:66 - Task 0.0:0 failed 4 times;
aborting job
2014-05-20 10:05:30 INFO DAGScheduler:50 - Failed to run count at
CountTool.java:44
Exception in thread "main" org.apache.spark.SparkException: Job aborted:
Task 0.0:0 failed 4 times (most recent failure: Exception failure:
java.io.EOFException)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
spark-shell also gives the same error:
scala> sc.textFile("/home/hadoop/scripts/clustering/deals.txt").count()
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server03.mydomain.com:35256/user/Executor#-1346794213]
with ID 2
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server02.mydomain.com:60855/user/Executor#-1316203186]
with ID 0
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server07.mydomain.com:47643/user/Executor#-556408193]
with ID 4
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server05.mydomain.com:55188/user/Executor#466888766]
with ID 5
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server06.mydomain.com:34672/user/Executor#299947202]
with ID 1
14/05/20 10:00:22 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server03.mydomain.com:57441 with 294.9 MB RAM
14/05/20 10:00:22 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server02.mydomain.com:55847 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server07.mydomain.com:52932 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server05.mydomain.com:45204 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server06.mydomain.com:57226 with 294.9 MB RAM
14/05/20 10:00:23 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkexecu...@server04.mydomain.com:37144/user/Executor#237933685]
with ID 3
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server04.mydomain.com:55928 with 294.9 MB RAM
14/05/20 10:00:24 INFO MemoryStore: ensureFreeSpace(78267) called with
curMem=0, maxMem=309225062
14/05/20 10:00:24 INFO MemoryStore: Block broadcast_0 stored as values to
memory (estimated size 76.4 KB, free 294.8 MB)
14/05/20 10:00:24 INFO FileInputFormat: Total input paths to process : 1
14/05/20 10:00:24 INFO SparkContext: Starting job: count at <console>:13
14/05/20 10:00:24 INFO DAGScheduler: Got job 0 (count at <console>:13) with
2 output partitions (allowLocal=false)
14/05/20 10:00:24 INFO DAGScheduler: Final stage: Stage 0 (count at
<console>:13)
14/05/20 10:00:24 INFO DAGScheduler: Parents of final stage: List()
14/05/20 10:00:24 INFO DAGScheduler: Missing parents: List()
14/05/20 10:00:24 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at
textFile at <console>:13), which has no missing parents
14/05/20 10:00:24 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0
(MappedRDD[1] at textFile at <console>:13)
14/05/20 10:00:24 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/05/20 10:00:24 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:24 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 5 ms
14/05/20 10:00:24 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on
executor 5: server05.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:24 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 0 (task 0.0:0)
14/05/20 10:00:25 WARN TaskSetManager: Loss was due to
java.io.FileNotFoundException
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:468)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:373)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:138)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:711)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:93)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:54)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:156)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 2 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 2 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 1]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 3 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 1 (task 0.0:1)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 2]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:1 as TID 4 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 3 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 3]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 5 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 4 (task 0.0:1)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 4]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:1 as TID 6 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 5 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 5]
14/05/20 10:00:25 ERROR TaskSetManager: Task 0.0:0 failed 4 times; aborting
job
14/05/20 10:00:25 INFO DAGScheduler: Failed to run count at <console>:13
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 6]
14/05/20 10:00:25 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
(most recent failure: Exception failure: java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
But I can load a hdfs file from scala-shell:
scala>
sc.textFile("hdfs://server01.mydomain.com:/user/hadoop/results/clustering/deals.txt").count()
14/05/20 10:02:57 INFO MemoryStore: ensureFreeSpace(111825) called with
curMem=681339, maxMem=309225062
14/05/20 10:02:57 INFO MemoryStore: Block broadcast_7 stored as values to
memory (estimated size 109.2 KB, free 294.1 MB)
14/05/20 10:02:57 INFO FileInputFormat: Total input paths to process : 1
14/05/20 10:02:57 INFO SparkContext: Starting job: count at <console>:13
14/05/20 10:02:57 INFO DAGScheduler: Got job 6 (count at <console>:13) with
2 output partitions (allowLocal=false)
14/05/20 10:02:57 INFO DAGScheduler: Final stage: Stage 6 (count at
<console>:13)
14/05/20 10:02:57 INFO DAGScheduler: Parents of final stage: List()
14/05/20 10:02:57 INFO DAGScheduler: Missing parents: List()
14/05/20 10:02:57 INFO DAGScheduler: Submitting Stage 6 (MappedRDD[15] at
textFile at <console>:13), which has no missing parents
14/05/20 10:02:57 INFO DAGScheduler: Submitting 2 missing tasks from Stage 6
(MappedRDD[15] at textFile at <console>:13)
14/05/20 10:02:57 INFO TaskSchedulerImpl: Adding task set 6.0 with 2 tasks
14/05/20 10:02:57 INFO TaskSetManager: Starting task 6.0:0 as TID 47 on
executor 2: server03.mydomain.com (NODE_LOCAL)
14/05/20 10:02:57 INFO TaskSetManager: Serialized task 6.0:0 as 1688 bytes
in 0 ms
14/05/20 10:02:57 INFO TaskSetManager: Starting task 6.0:1 as TID 48 on
executor 5: server05.mydomain.com (NODE_LOCAL)
14/05/20 10:02:57 INFO TaskSetManager: Serialized task 6.0:1 as 1688 bytes
in 1 ms
14/05/20 10:02:58 INFO DAGScheduler: Completed ResultTask(6, 0)
14/05/20 10:02:58 INFO TaskSetManager: Finished TID 47 in 454 ms on
server03.mydomain.com (progress: 1/2)
14/05/20 10:02:58 INFO TaskSetManager: Finished TID 48 in 504 ms on
server05.mydomain.com (progress: 2/2)
14/05/20 10:02:58 INFO DAGScheduler: Completed ResultTask(6, 1)
14/05/20 10:02:58 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks
have all completed, from pool
14/05/20 10:02:58 INFO DAGScheduler: Stage 6 (count at <console>:13)
finished in 0.506 s
14/05/20 10:02:58 INFO SparkContext: Job finished: count at <console>:13,
took 0.513880747 s
res8: Long = 101
I also tried to load file into an ArrayList and use parallelize(), it worked
too. But I don't understand why I can't just load a file and count it? Could
someone please help me understand the problem?
Thanks in advance...
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-loading-files-Loss-was-due-to-java-io-EOFException-java-io-EOFException-tp6090.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
