In the past I have seen this happen when I filled up HDFS and some core nodes became unhealthy. There was no longer anywhere to replicate the data. >From your command it looks like you should have 1 master and 2 core nodes in your cluster. Can you verify both the core nodes are healthy?
On Wed, Mar 2, 2016 at 6:01 AM, Oleg Ruchovets <oruchov...@gmail.com> wrote: > Here is my command: > aws emr create-cluster --release-label emr-4.3.0 --name "ClusterJava8" > --use-default-roles --applications Name=Ganglia Name=Hive Name=Hue > Name=Mahout Name=Pig Name=Spark --ec2-attributes KeyName=CC-ES-Demo > --instance-count 3 --instance-type m3.xlarge --use-default-roles > --bootstrap-action Path=s3://crayon-emr-scripts/emr_java_8.sh > > I am using bootstrap script to install java 8. > > When I choose applications (Name=Ganglia Name=Hive Name=Hue Name=Mahout > Name=Pig Name=Spark) problem is gone. I fixed on the way Lzo not found > exception. Now I have another problem that I have no idea why it happens: > I tries to copy file to hdfs and got this exception (file is very small , > just couple of kb). > > > > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /input/test.txt._COPYING_ could only be replicated to 0 nodes instead of > minReplication (=1). There are 0 datanode(s) running and no node(s) are > excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) > > at org.apache.hadoop.ipc.Client.call(Client.java:1476) > at org.apache.hadoop.ipc.Client.call(Client.java:1407) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) > at com.sun.proxy.$Proxy9.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy10.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) > put: File /input/test.txt._COPYING_ could only be replicated to 0 nodes > instead of minReplication (=1). There are 0 datanode(s) running and no > node(s) are excluded in this operation. > > > On Wed, Mar 2, 2016 at 4:09 AM, Gourav Sengupta <gourav.sengu...@gmail.com > > wrote: > >> Hi, >> >> which region are you using the EMR clusters from? Is there any tweaking >> of the HADOOP parameters that you are doing before starting the clusters? >> >> If you are using AWS CLI to start the cluster just send across the >> command. >> >> I have, never till date, faced any such issues in the Ireland region. >> >> >> Regards, >> Gourav Sengupta >> >> On Tue, Mar 1, 2016 at 9:15 AM, Oleg Ruchovets <oruchov...@gmail.com> >> wrote: >> >>> Hi , I am installed EMR 4.3.0 with spark. I tries to enter spark shell >>> but it looks it does't work and throws exceptions. >>> Please advice: >>> >>> [hadoop@ip-172-31-39-37 conf]$ cd /usr/bin/ >>> [hadoop@ip-172-31-39-37 bin]$ ./spark-shell >>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; >>> support was removed in 8.0 >>> 16/03/01 09:11:48 INFO SecurityManager: Changing view acls to: hadoop >>> 16/03/01 09:11:48 INFO SecurityManager: Changing modify acls to: hadoop >>> 16/03/01 09:11:48 INFO SecurityManager: SecurityManager: authentication >>> disabled; ui acls disabled; users with view permissions: Set(hadoop); users >>> with modify permissions: Set(hadoop) >>> 16/03/01 09:11:49 INFO HttpServer: Starting HTTP Server >>> 16/03/01 09:11:49 INFO Utils: Successfully started service 'HTTP class >>> server' on port 47223. >>> Welcome to >>> ____ __ >>> / __/__ ___ _____/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>> /_/ >>> >>> Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.8.0_71) >>> Type in expressions to have them evaluated. >>> Type :help for more information. >>> 16/03/01 09:11:53 INFO SparkContext: Running Spark version 1.6.0 >>> 16/03/01 09:11:53 INFO SecurityManager: Changing view acls to: hadoop >>> 16/03/01 09:11:53 INFO SecurityManager: Changing modify acls to: hadoop >>> 16/03/01 09:11:53 INFO SecurityManager: SecurityManager: authentication >>> disabled; ui acls disabled; users with view permissions: Set(hadoop); users >>> with modify permissions: Set(hadoop) >>> 16/03/01 09:11:54 INFO Utils: Successfully started service 'sparkDriver' >>> on port 52143. >>> 16/03/01 09:11:54 INFO Slf4jLogger: Slf4jLogger started >>> 16/03/01 09:11:54 INFO Remoting: Starting remoting >>> 16/03/01 09:11:54 INFO Remoting: Remoting started; listening on >>> addresses :[akka.tcp://sparkDriverActorSystem@172.31.39.37:42989] >>> 16/03/01 09:11:54 INFO Utils: Successfully started service >>> 'sparkDriverActorSystem' on port 42989. >>> 16/03/01 09:11:54 INFO SparkEnv: Registering MapOutputTracker >>> 16/03/01 09:11:54 INFO SparkEnv: Registering BlockManagerMaster >>> 16/03/01 09:11:54 INFO DiskBlockManager: Created local directory at >>> /mnt/tmp/blockmgr-afaf0e7f-086e-49f1-946d-798e605a3fdc >>> 16/03/01 09:11:54 INFO MemoryStore: MemoryStore started with capacity >>> 518.1 MB >>> 16/03/01 09:11:55 INFO SparkEnv: Registering OutputCommitCoordinator >>> 16/03/01 09:11:55 INFO Utils: Successfully started service 'SparkUI' on >>> port 4040. >>> 16/03/01 09:11:55 INFO SparkUI: Started SparkUI at >>> http://172.31.39.37:4040 >>> 16/03/01 09:11:55 INFO RMProxy: Connecting to ResourceManager at / >>> 172.31.39.37:8032 >>> 16/03/01 09:11:55 INFO Client: Requesting a new application from cluster >>> with 2 NodeManagers >>> 16/03/01 09:11:55 INFO Client: Verifying our application has not >>> requested more than the maximum memory capability of the cluster (11520 MB >>> per container) >>> 16/03/01 09:11:55 INFO Client: Will allocate AM container, with 896 MB >>> memory including 384 MB overhead >>> 16/03/01 09:11:55 INFO Client: Setting up container launch context for >>> our AM >>> 16/03/01 09:11:55 INFO Client: Setting up the launch environment for our >>> AM container >>> 16/03/01 09:11:55 INFO Client: Preparing resources for our AM container >>> 16/03/01 09:11:56 INFO Client: Uploading resource >>> file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar -> >>> hdfs:// >>> 172.31.39.37:8020/user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >>> 16/03/01 09:11:56 INFO MetricsSaver: MetricsConfigRecord >>> disabledInCluster: false instanceEngineCycleSec: 60 clusterEngineCycleSec: >>> 60 disableClusterEngine: false maxMemoryMb: 3072 maxInstanceCount: 500 >>> lastModified: 1456818856695 >>> 16/03/01 09:11:56 INFO MetricsSaver: Created MetricsSaver >>> j-2FT6QNFSPTHNX:i-5f6bcadb:SparkSubmit:04807 period:60 >>> /mnt/var/em/raw/i-5f6bcadb_20160301_SparkSubmit_04807_raw.bin >>> 16/03/01 09:11:56 WARN DFSClient: DataStreamer Exception >>> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >>> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >>> could only be replicated to 0 nodes instead of minReplication (=1). There >>> are 0 datanode(s) running and no node(s) are excluded in this operation. >>> at >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >>> at >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >>> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >>> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >>> 16/03/01 09:11:56 INFO Client: Deleting staging directory >>> .sparkStaging/application_1456818849676_0005 >>> 16/03/01 09:11:56 ERROR SparkContext: Error initializing SparkContext. >>> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >>> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >>> could only be replicated to 0 nodes instead of minReplication (=1). There >>> are 0 datanode(s) running and no node(s) are excluded in this operation. >>> at >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >>> at >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >>> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >>> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >>> 16/03/01 09:11:56 INFO SparkUI: Stopped Spark web UI at >>> http://172.31.39.37:4040 >>> 16/03/01 09:11:56 INFO YarnClientSchedulerBackend: Stopped >>> 16/03/01 09:11:56 INFO MapOutputTrackerMasterEndpoint: >>> MapOutputTrackerMasterEndpoint stopped! >>> 16/03/01 09:11:56 ERROR Utils: Uncaught exception in thread main >>> java.lang.NullPointerException >>> at >>> org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152) >>> at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1231) >>> at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96) >>> at >>> org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1756) >>> at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1229) >>> at org.apache.spark.SparkContext.stop(SparkContext.scala:1755) >>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:602) >>> at >>> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) >>> at $line3.$read$$iwC$$iwC.<init>(<console>:15) >>> at $line3.$read$$iwC.<init>(<console>:24) >>> at $line3.$read.<init>(<console>:26) >>> at $line3.$read$.<init>(<console>:30) >>> at $line3.$read$.<clinit>(<console>) >>> at $line3.$eval$.<init>(<console>:7) >>> at $line3.$eval$.<clinit>(<console>) >>> at $line3.$eval.$print(<console>) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >>> at >>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) >>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >>> at >>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >>> at >>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >>> at >>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125) >>> at >>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) >>> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) >>> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) >>> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) >>> at >>> org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> at >>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >>> at org.apache.spark.repl.SparkILoop.org >>> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >>> at org.apache.spark.repl.Main$.main(Main.scala:31) >>> at org.apache.spark.repl.Main.main(Main.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> 16/03/01 09:11:56 INFO SparkContext: Successfully stopped SparkContext >>> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File >>> /user/hadoop/.sparkStaging/application_1456818849676_0005/spark-assembly-1.6.0-hadoop2.7.1-amzn-0.jar >>> could only be replicated to 0 nodes instead of minReplication (=1). There >>> are 0 datanode(s) running and no node(s) are excluded in this operation. >>> at >>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) >>> at >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:632) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1407) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:238) >>> at com.sun.proxy.$Proxy16.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >>> at com.sun.proxy.$Proxy17.addBlock(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1441) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >>> at >>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) >>> >>> java.lang.NullPointerException >>> at >>> org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1367) >>> at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> at >>> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028) >>> at $iwC$$iwC.<init>(<console>:15) >>> at $iwC.<init>(<console>:24) >>> at <init>(<console>:26) >>> at .<init>(<console>:30) >>> at .<clinit>(<console>) >>> at .<init>(<console>:7) >>> at .<clinit>(<console>) >>> at $print(<console>) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >>> at >>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) >>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >>> at >>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >>> at >>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >>> at >>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132) >>> at >>> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) >>> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) >>> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) >>> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) >>> at >>> org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> at >>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >>> at org.apache.spark.repl.SparkILoop.org >>> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >>> at org.apache.spark.repl.Main$.main(Main.scala:31) >>> at org.apache.spark.repl.Main.main(Main.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> <console>:16: error: not found: value sqlContext >>> import sqlContext.implicits._ >>> ^ >>> <console>:16: error: not found: value sqlContext >>> import sqlContext.sql >>> ^ >>> >>> >> >
