Error executing job in YARN. Exception Shell$ExitCodeException
I'm trying to execute a work in YARN but I get an error. I have checked the yarn-site.xml, the classpath seems to be alright. I read https://issues.apache.org/jira/browse/YARN-1473 as well, but it didn't work to me either. Container for appattempt_1413373500815_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:511) at org.apache.hadoop.util.Shell.run(Shell.java:424) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) What else could it be happened? AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY WARNING.\nThis message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, reta in or redistribute any portion of this E-mail.
Trying MapReduce with MRUnit and MultipleOutput.
I'm trying to test some MapReduces with MRUnit 1.1.0, but I didn't get results. The code that I execute is: mapTextDriver.withInput(new LongWritable(1), new Text(content)); ListPairNullWritable, Text outputs = mapTextDriver.run(); But, I never got an output, the list always has size 0. I was reading the JIRA https://issues.apache.org/jira/browse/MRUNIT-13 where they add the new feature to MRUnit but I don't know what I'm missing.. I included the annotations: @RunWith(..) @PrepareForTest(..) I tried as well to execute the test like: mapTextDriver.runTest(); and it works. But I like more to get a list of result and analyze them. AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY WARNING.\nThis message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, reta in or redistribute any portion of this E-mail.
How to know the number of nodes?
I would like to know the number of nodes my cluster has. Is it possible in YARN and MapReduce from Java? I suppose that you have installed a TaskTracker in any node. AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY WARNING.\nThis message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, reta in or redistribute any portion of this E-mail.
How to get the max number of reducers in Yarn
I have been working with MapReduce1, (JobTracker and TaskTrakers). Some of my jobs I want to define the number of reduces to the maximum capacity of my cluster. I did it with this: int max = new JobClient(new JobConf(jConf)).getClusterStatus().getMaxReduceTasks(); Job job = new Job(jConf, this.getClass().getName()); job.setNumReduceTasks(max); Now, I want to work with YARN and it seems that it doesn't work. I think that YARN manages the number of reducers in real time depending of the resources it has available. The method getMaxReduceTasks it returns me just two. don't know if there's another way to set the number the reducer to the real capacity of the cluster or what I'm doing wrong. I guess that if I don't use setNumReduceTaskm, it'll get one because the default value. AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY WARNING.\nThis message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, reta in or redistribute any portion of this E-mail.
