Re: Versions of Jetty Log4j in CDHu3
Oh ok. Thanks Chaidy. I was wondering if I can just use log4j compression facility along with timebased rolling as in terms of gzip and then use less disk space. This seems to be feature available in 1.3 (not sure if it is also available in log4j-1.2.15), I think I need to give it a try and see. But on the other hand, if it does not work, any process suggestion that I need to follow to upgrade log4j 1.3 and expect hadoop to be compatible with new log4j library (version changes?) Thanks, Nikhil On Sun, Apr 29, 2012 at 11:07 AM, CHAIDY cha...@nsccsz.gov.cn wrote: Hi, Nikhil! FYI: jetty-6.1.26, log4j-1.2.15 At 2012-04-29 13:03:44,Nikhil mnik...@gmail.com wrote: Hi, I was wondering about the release versions of both Jetty and log4j components released as part of CDHu3 release package. Can someone please let me know. Thanks.
Re: Can’t stop hadoop daemons
Hey Barry, How did you start these daemons in the first place? On Sun, Apr 29, 2012 at 1:16 AM, Barry, Sean F sean.f.ba...@intel.com wrote: hduser@master:~ /usr/java/jdk1.7.0/bin/jps 20907 TaskTracker 20629 SecondaryNameNode 25863 Jps 20777 JobTracker 20383 NameNode 20507 DataNode hduser@master:~ stop- stop-all.sh stop-balancer.sh stop-dfs.sh stop-mapred.sh hduser@master:~ stop-all.sh no jobtracker to stop master: no tasktracker to stop slave: no tasktracker to stop no namenode to stop master: no datanode to stop slave: no datanode to stop master: no secondarynamenode to stop hduser@master:~ as you can see jps shows that the daemons are running but I cant stop them with the stop-all.sh command. Does anyone have an idea for why this is happening ? -SB -- Harsh J
RE: Can’t stop hadoop daemons
I just restarted my machines and it works fine now. -SB -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Sunday, April 29, 2012 5:55 AM To: common-user@hadoop.apache.org Subject: Re: Can’t stop hadoop daemons Hey Barry, How did you start these daemons in the first place? On Sun, Apr 29, 2012 at 1:16 AM, Barry, Sean F sean.f.ba...@intel.com wrote: hduser@master:~ /usr/java/jdk1.7.0/bin/jps 20907 TaskTracker 20629 SecondaryNameNode 25863 Jps 20777 JobTracker 20383 NameNode 20507 DataNode hduser@master:~ stop- stop-all.sh stop-balancer.sh stop-dfs.sh stop-mapred.sh hduser@master:~ stop-all.sh no jobtracker to stop master: no tasktracker to stop slave: no tasktracker to stop no namenode to stop master: no datanode to stop slave: no datanode to stop master: no secondarynamenode to stop hduser@master:~ as you can see jps shows that the daemons are running but I cant stop them with the stop-all.sh command. Does anyone have an idea for why this is happening ? -SB -- Harsh J
Re: Can’t stop hadoop daemons
Sean, One cause I can think of is that your PID directory is on /tmp or so, and the original saved PID files got cleared away by tmpwatch, leading to this state. To fix such a flaw, export HADOOP_PID_DIR in hadoop-env.sh to a more persistent location (such as within HADOOP_HOME/pids itself, say). What version of Hadoop are you using though? On Mon, Apr 30, 2012 at 12:58 AM, Barry, Sean F sean.f.ba...@intel.com wrote: I just restarted my machines and it works fine now. -SB -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Sunday, April 29, 2012 5:55 AM To: common-user@hadoop.apache.org Subject: Re: Can’t stop hadoop daemons Hey Barry, How did you start these daemons in the first place? On Sun, Apr 29, 2012 at 1:16 AM, Barry, Sean F sean.f.ba...@intel.com wrote: hduser@master:~ /usr/java/jdk1.7.0/bin/jps 20907 TaskTracker 20629 SecondaryNameNode 25863 Jps 20777 JobTracker 20383 NameNode 20507 DataNode hduser@master:~ stop- stop-all.sh stop-balancer.sh stop-dfs.sh stop-mapred.sh hduser@master:~ stop-all.sh no jobtracker to stop master: no tasktracker to stop slave: no tasktracker to stop no namenode to stop master: no datanode to stop slave: no datanode to stop master: no secondarynamenode to stop hduser@master:~ as you can see jps shows that the daemons are running but I cant stop them with the stop-all.sh command. Does anyone have an idea for why this is happening ? -SB -- Harsh J -- Harsh J
Nightly Builds
Hi everyone, i'm want to run hadoop (hbase) in an IBM JVM. I've seen that there were several patches for that reason. I am not a developer so my knowleges in building java jars fromsources are very limited and the link with the nightly builds do not work. I only need hadoop-core-1.0.3.jar. Where can i find it even when it's unstable? Thanks and bye, Stephan
Re: DFSClient error
It sounds to me like you're running out of DN xceivers. Try the solution offered at http://hbase.apache.org/book.html#dfs.datanode.max.xcievers I.e., add: property namedfs.datanode.max.xcievers/name value4096/value /property To your DNs' config/hdfs-site.xml and restart the DNs. On Mon, Apr 30, 2012 at 1:35 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I even tried to lower number of parallel jobs even further but I still get these errors. Any suggestion on how to troubleshoot this issue would be very helpful. Should I run hadoop fsck? How do people troubleshoot such issues?? Does it sound like a bug? 2012-04-27 14:37:42,921 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2012-04-27 14:37:42,931 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.199:50010java.io.EOFException 2012-04-27 14:37:42,932 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_6343044536824463287_24619 2012-04-27 14:37:42,932 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.199:50010 2012-04-27 14:37:42,935 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException 2012-04-27 14:37:42,935 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_2837215798109471362_24620 2012-04-27 14:37:42,936 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.204:50010 2012-04-27 14:37:42,937 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2012-04-27 14:37:42,939 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException 2012-04-27 14:37:42,939 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_2223489090936415027_24620 2012-04-27 14:37:42,940 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.198:50010 2012-04-27 14:37:42,943 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException 2012-04-27 14:37:42,943 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_1265169201875643059_24620 2012-04-27 14:37:42,944 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.197:50010 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3446) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822) 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - Error Recovery for block blk_1265169201875643059_24620 bad datanode[0] nodes == null 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - Could not get block locations. Source file /tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411/job.jar - Aborting... 2012-04-27 14:37:42,945 [Thread-4] INFO org.apache.hadoop.mapred.JobClient - Cleaning up the staging area hdfs://dsdb1:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411 2012-04-27 14:37:42,945 [Thread-4] ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.EOFException 2012-04-27 14:37:42,996 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.200:50010java.io.IOException: Bad connect ack with firstBadLink as 125.18.62.198:50010 2012-04-27 14:37:42,996 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_-7583284266913502018_24621 2012-04-27 14:37:42,997 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException 2012-04-27 14:37:42,997 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_4207260385919079785_24622 2012-04-27 14:37:42,998 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.198:50010 2012-04-27 14:37:43,000 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.198:50010 2012-04-27 14:37:43,002 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException 2012-04-27 14:37:43,002 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_-2859304645525022496_24624 2012-04-27 14:37:43,003 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode
Nightly Build
Hi everyone, i'm want to run hadoop (hbase) in an IBM JVM. I've seen that there were several patches for that reason. I am not a developer so my knowleges in building java jars fromsources are very limited and the link with the nightly builds do not work. I only need hadoop-core-1.0.3.jar. Where can i find it even when it's unstable? Thanks and bye, Stephan
Re: DFSClient error
Thanks for the quick response, appreciate it. It looks like this might be the issue. But I am still trying to understand what is causing so many threads in my situation? Is this thread per block that gets created or per file? Because if it's per file then it should not be more than 15. My second question, I read around 5 .gz files in 5 separate processed. This is constant and also the size of those 5 is roughly equivalent. So then why does it fail only halfway and not right in the begining. I am reading around 400 files and it always fails when I reach around 180th file. What's the default value of xceivers? Is 4096 consume too much of stack size? Thanks On Sun, Apr 29, 2012 at 1:14 PM, Harsh J ha...@cloudera.com wrote: It sounds to me like you're running out of DN xceivers. Try the solution offered at http://hbase.apache.org/book.html#dfs.datanode.max.xcievers I.e., add: property namedfs.datanode.max.xcievers/name value4096/value /property To your DNs' config/hdfs-site.xml and restart the DNs. On Mon, Apr 30, 2012 at 1:35 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I even tried to lower number of parallel jobs even further but I still get these errors. Any suggestion on how to troubleshoot this issue would be very helpful. Should I run hadoop fsck? How do people troubleshoot such issues?? Does it sound like a bug? 2012-04-27 14:37:42,921 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2012-04-27 14:37:42,931 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.199:50010 java.io.EOFException 2012-04-27 14:37:42,932 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_6343044536824463287_24619 2012-04-27 14:37:42,932 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.199:50010 2012-04-27 14:37:42,935 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.204:50010 java.io.EOFException 2012-04-27 14:37:42,935 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_2837215798109471362_24620 2012-04-27 14:37:42,936 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.204:50010 2012-04-27 14:37:42,937 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2012-04-27 14:37:42,939 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.198:50010 java.io.EOFException 2012-04-27 14:37:42,939 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_2223489090936415027_24620 2012-04-27 14:37:42,940 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.198:50010 2012-04-27 14:37:42,943 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.197:50010 java.io.EOFException 2012-04-27 14:37:42,943 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_1265169201875643059_24620 2012-04-27 14:37:42,944 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Excluding datanode 125.18.62.197:50010 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3446) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822) 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - Error Recovery for block blk_1265169201875643059_24620 bad datanode[0] nodes == null 2012-04-27 14:37:42,945 [Thread-5] WARN org.apache.hadoop.hdfs.DFSClient - Could not get block locations. Source file /tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411/job.jar - Aborting... 2012-04-27 14:37:42,945 [Thread-4] INFO org.apache.hadoop.mapred.JobClient - Cleaning up the staging area hdfs://dsdb1:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411 2012-04-27 14:37:42,945 [Thread-4] ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.EOFException 2012-04-27 14:37:42,996 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Exception in createBlockOutputStream 125.18.62.200:50010java.io.IOException: Bad connect ack with firstBadLink as 125.18.62.198:50010 2012-04-27 14:37:42,996 [Thread-5] INFO org.apache.hadoop.hdfs.DFSClient - Abandoning block blk_-7583284266913502018_24621 2012-04-27 14:37:42,997 [Thread-5] INFO
Getting job progress in java application
Hello I'd like to ask you what is the preferred way of getting running jobs progress from Java application, that has executed them. Im using Hadoop 0.20.203, tried job.end.notification.url property that works well, but as the property name says, it sends only job end notifications. What I need is to get updates on map() and reduce() progress. Please help how to do this. Thanks. Ondrej Klimpera
Hadoop Security 2011/2012
Hi Hadoop users, Has anyone attended a Hadoop conference where there were talks about any new features in Hadoop security ? I am trying to figure out if any new features have been added to Hadoop security after Kerberos . Thanks -- Cheers Atul
Re: Getting job progress in java application
Take a look at the JobClient API. You can use that to get the current progress of a running job. On Sunday, April 29, 2012, Ondřej Klimpera wrote: Hello I'd like to ask you what is the preferred way of getting running jobs progress from Java application, that has executed them. Im using Hadoop 0.20.203, tried job.end.notification.url property that works well, but as the property name says, it sends only job end notifications. What I need is to get updates on map() and reduce() progress. Please help how to do this. Thanks. Ondrej Klimpera -- *Note that I'm no longer using my Yahoo! email address. Please email me at billgra...@gmail.com going forward.*
Mass SocketTimeoutException - 0.20.203
Tons of errors seen after Map 100% Reduce 50%, but the job still struggles to finish. What is the possible reason? Is this issue fixed in any of the version? java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.105.19.35:58539 remote=/10.105.19.35:50010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTim eout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream .java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream .java:128) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream .java:116) at java.io.DataInputStream.readShort(DataInputStream.java:295) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOu tputStream(DFSClient.java:3060) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutp utStream(DFSClient.java:2983) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(D FSClient.java:2255) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer. run(DFSClient.java:2446)
EMR Hadoop
Hi guys : 1) Does anybody know if there is a VM out there which runs EMR hadoop ? I would like to have a local vm for dev purposes that mirrored the EMR hadoop instances. 2) How does EMR's hadoop differ from apache hadoop and Cloudera's hadoop ? -- Jay Vyas MMSB/UCHC
