Hadoop 0.21
Hi All, According to the Hadoop release notes, version 0.21.0 should not be considered stable or suitable for production: 23 August, 2010: release 0.21.0 available This release contains many improvements, new features, bug fixes and optimizations. It has not undergone testing at scale and should not be considered stable or suitable for production. This release is being classified as a minor release, which means that it should be API compatible with 0.20.2. Is this still the case ? Thank you, Saurabh
Re: Hadoop 0.21
Yep. J-D On Tue, Dec 6, 2011 at 10:41 AM, Saurabh Sehgal saurabh@gmail.com wrote: Hi All, According to the Hadoop release notes, version 0.21.0 should not be considered stable or suitable for production: 23 August, 2010: release 0.21.0 available This release contains many improvements, new features, bug fixes and optimizations. It has not undergone testing at scale and should not be considered stable or suitable for production. This release is being classified as a minor release, which means that it should be API compatible with 0.20.2. Is this still the case ? Thank you, Saurabh
Re: Hadoop 0.21
I second Vinod´s idea. Get the latest stable from Cloudera. Their binaries are near perfect! On Tue, Dec 6, 2011 at 1:46 PM, T Vinod Gupta tvi...@readypulse.com wrote: Saurabh, Its best if you go through the hbase book - Lars George's book HBase the Definitive Guide. Your best bet is to build all binaries yourself or get a stable build from cloudera. I was in this situation few months ago and had to spend a lot of time before I was able to get a production ready hbase version up and running. thanks vinod On Tue, Dec 6, 2011 at 10:41 AM, Saurabh Sehgal saurabh@gmail.com wrote: Hi All, According to the Hadoop release notes, version 0.21.0 should not be considered stable or suitable for production: 23 August, 2010: release 0.21.0 available This release contains many improvements, new features, bug fixes and optimizations. It has not undergone testing at scale and should not be considered stable or suitable for production. This release is being classified as a minor release, which means that it should be API compatible with 0.20.2. Is this still the case ? Thank you, Saurabh -- --- Get your facts first, then you can distort them as you please.--
Change block size from 64M to 128M does not work on Hadoop-0.21
Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use fsck to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Your client (put) machine must have the same block size configuration during upload as well. Alternatively, you may do something explicit like `hadoop dfs -Ddfs.block.size=size -put file file` On Thu, May 5, 2011 at 12:59 AM, He Chen airb...@gmail.com wrote: Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use fsck to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen -- Harsh J
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Hi Harsh Thank you for the reply. Actually, the hadoop directory is on my NFS server, every node reads the same file from NFS server. I think this is not a problem. I like your second solution. But I am not sure, whether the namenode will divide those 128MB blocks to smaller ones in future or not. Chen On Wed, May 4, 2011 at 3:00 PM, Harsh J ha...@cloudera.com wrote: Your client (put) machine must have the same block size configuration during upload as well. Alternatively, you may do something explicit like `hadoop dfs -Ddfs.block.size=size -put file file` On Thu, May 5, 2011 at 12:59 AM, He Chen airb...@gmail.com wrote: Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use fsck to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen -- Harsh J
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Tried second solution. Does not work, still 2 64M blocks. h On Wed, May 4, 2011 at 3:16 PM, He Chen airb...@gmail.com wrote: Hi Harsh Thank you for the reply. Actually, the hadoop directory is on my NFS server, every node reads the same file from NFS server. I think this is not a problem. I like your second solution. But I am not sure, whether the namenode will divide those 128MB blocks to smaller ones in future or not. Chen On Wed, May 4, 2011 at 3:00 PM, Harsh J ha...@cloudera.com wrote: Your client (put) machine must have the same block size configuration during upload as well. Alternatively, you may do something explicit like `hadoop dfs -Ddfs.block.size=size -put file file` On Thu, May 5, 2011 at 12:59 AM, He Chen airb...@gmail.com wrote: Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use fsck to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen -- Harsh J
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Got it. Thankyou Harsh. BTW It is `hadoop dfs -Ddfs.blocksize=size -put file file`. No dot between block and size On Wed, May 4, 2011 at 3:18 PM, He Chen airb...@gmail.com wrote: Tried second solution. Does not work, still 2 64M blocks. h On Wed, May 4, 2011 at 3:16 PM, He Chen airb...@gmail.com wrote: Hi Harsh Thank you for the reply. Actually, the hadoop directory is on my NFS server, every node reads the same file from NFS server. I think this is not a problem. I like your second solution. But I am not sure, whether the namenode will divide those 128MB blocks to smaller ones in future or not. Chen On Wed, May 4, 2011 at 3:00 PM, Harsh J ha...@cloudera.com wrote: Your client (put) machine must have the same block size configuration during upload as well. Alternatively, you may do something explicit like `hadoop dfs -Ddfs.block.size=size -put file file` On Thu, May 5, 2011 at 12:59 AM, He Chen airb...@gmail.com wrote: Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use fsck to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen -- Harsh J
Hadoop 0.21 running problems , no namenode to stop
Problems running local installation of hadoop on single-node cluster
I followed instructions given by tutorials to run hadoop-0.21 on a single node
cluster.
The first problem I encountered was that of HADOOP-6953. Thankfully that has
got fixed.
The other problem I am facing is that the datanode does not start. This I guess
because when I run stop-dfs.sh for datanode, I get a message
no datanode to stop
I am wondering if it is related remotely to the difference in the IP addresses
on my computer
127.0.0.1 localhost
127.0.1.1 my-laptop
Although I am aware of this, I do not know how to fix this.
I am unable to even run a simple pi estimate example on the haddop installation
This is the output I get is
bin/hadoop jar hadoop-mapred-examples-0.21.0.jar pi 10 10
Number of Maps = 10
Samples per Map = 10
11/03/02 23:38:47 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30
And nothing else for long long time.
I have not set the dfs.namedir and dfs.datadir in my hdfs-site.xml. But After
running bin/hadoop namenode -format, I see that the tmp.dir has a folder with
dfs/data and dfs/data folders for the two directories.
what Am I doing wrong? Any help is appreciated.
Here are my configuration files
Regards,
Shivani
hdfs-site.xml
property
namedfs.replication/name
value1/value
descriptionDefault block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
/description
/property
core-site.xml
property
namehadoop.tmp.dir/name
value/usr/local/hadoop-${user.name}/value
descriptionA base for other temporary directories./description
/property
property
namefs.default.name/name
valuehdfs://localhost:54310/value
descriptionThe name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem./description
/property
mapred-site.xml
property
namemapred.job.tracker/name
valuelocalhost:54311/value
descriptionThe host and port that the MapReduce job tracker runs
at. If local, then jobs are run in-process as a single map
and reduce task.
/description
/property
Re: Hadoop 0.21 running problems , no namenode to stop
Hi,
Please check logs, there might be some error occured while starting daemons
Please post the error
On Thu, Mar 3, 2011 at 10:24 AM, Shivani Rao sg...@purdue.edu wrote:
Problems running local installation of hadoop on single-node cluster
I followed instructions given by tutorials to run hadoop-0.21 on a single
node cluster.
The first problem I encountered was that of HADOOP-6953. Thankfully that
has got fixed.
The other problem I am facing is that the datanode does not start. This I
guess because when I run stop-dfs.sh for datanode, I get a message
no datanode to stop
I am wondering if it is related remotely to the difference in the IP
addresses on my computer
127.0.0.1 localhost
127.0.1.1 my-laptop
Although I am aware of this, I do not know how to fix this.
I am unable to even run a simple pi estimate example on the haddop
installation
This is the output I get is
bin/hadoop jar hadoop-mapred-examples-0.21.0.jar pi 10 10
Number of Maps = 10
Samples per Map = 10
11/03/02 23:38:47 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=30
And nothing else for long long time.
I have not set the dfs.namedir and dfs.datadir in my hdfs-site.xml. But
After running bin/hadoop namenode -format, I see that the tmp.dir has a
folder with dfs/data and dfs/data folders for the two directories.
what Am I doing wrong? Any help is appreciated.
Here are my configuration files
Regards,
Shivani
hdfs-site.xml
property
namedfs.replication/name
value1/value
descriptionDefault block replication.
The actual number of replications can be specified when the file is
created.
The default is used if replication is not specified in create time.
/description
/property
core-site.xml
property
namehadoop.tmp.dir/name
value/usr/local/hadoop-${user.name}/value
descriptionA base for other temporary directories./description
/property
property
namefs.default.name/name
valuehdfs://localhost:54310/value
descriptionThe name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem./description
/property
mapred-site.xml
property
namemapred.job.tracker/name
valuelocalhost:54311/value
descriptionThe host and port that the MapReduce job tracker runs
at. If local, then jobs are run in-process as a single map
and reduce task.
/description
/property
Re:Re: Problem with building hadoop 0.21
Hi.Simon: I modified some coed related to scheduler and designed a customized scheduler .when I built the modified code, then the problems described above came up with it. I doubt whether there was something with my code, but after I built the out-of-box code, the same problems still existed. Can you tell me how to build and deploy a customized hadoop? Thank you! zhutao At 2011-02-28 11:21:16，Simon gsmst...@gmail.com wrote: Hey, Can you let us know why you want to replace all the jar files? That usually does not work, especially for development code in the code base. So, just use the one you have successfully compiled, don't replace jar files. Hope it can work. Simon 2011/2/27 朱韬 ryanzhu...@163.com Hi，guys: I checked out the source code fromhttp:// svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using this script: #!/bin/bash export JAVA_HOME=/usr/share/jdk1.6.0_14 export CFLAGS=-m64 export CXXFLAGS=-m64 export ANT_HOME=/opt/apache-ant-1.8.2 export PATH=$PATH:$ANT_HOME/bin ant -Dversion=0.21.0 -Dcompile.native=true -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar It was Ok before these steps. Then I replaced hadoop-mapred-0.21.0.jar, hadoop-mapred-0.21.0-sources.jar, hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files from the above step. Also I added my scheduler to lib. When starting the customed hadoop, I encountered the problems as blow: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/security/RefreshUserMappingsProtocol at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out 10.61.0.143: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out 10.61.0.7: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out 10.61.0.6: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.6: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.6: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.6: at java.security.AccessController.doPrivileged(Native Method) 10.61.0.6: at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.6: at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.6: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.143: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.143: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.143:at java.security.AccessController.doPrivileged(Native Method) 10.61.0.143:at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.143:at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.143:at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.143: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.7: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.7: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.7: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.7: at
Re: Re: Problem with building hadoop 0.21
I mean can you just make changes to the 0.21 version of your hadoop rather than put the 0.21 version jars to the latest code. There might be API breakdowns. Or you can try downloading source code of version 0.21 and try your steps. Thanks Simon 2011/2/28 朱韬 ryanzhu...@163.com Hi.Simon: I modified some coed related to scheduler and designed a customized scheduler .when I built the modified code, then the problems described above came up with it. I doubt whether there was something with my code, but after I built the out-of-box code, the same problems still existed. Can you tell me how to build and deploy a customized hadoop? Thank you! zhutao At 2011-02-28 11:21:16，Simon gsmst...@gmail.com wrote: Hey, Can you let us know why you want to replace all the jar files? That usually does not work, especially for development code in the code base. So, just use the one you have successfully compiled, don't replace jar files. Hope it can work. Simon 2011/2/27 朱韬 ryanzhu...@163.com Hi，guys: I checked out the source code fromhttp:// svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using this script: #!/bin/bash export JAVA_HOME=/usr/share/jdk1.6.0_14 export CFLAGS=-m64 export CXXFLAGS=-m64 export ANT_HOME=/opt/apache-ant-1.8.2 export PATH=$PATH:$ANT_HOME/bin ant -Dversion=0.21.0 -Dcompile.native=true -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar It was Ok before these steps. Then I replaced hadoop-mapred-0.21.0.jar, hadoop-mapred-0.21.0-sources.jar, hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files from the above step. Also I added my scheduler to lib. When starting the customed hadoop, I encountered the problems as blow: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/security/RefreshUserMappingsProtocol at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out 10.61.0.143: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out 10.61.0.7: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out 10.61.0.6: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.6: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.6: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.6: at java.security.AccessController.doPrivileged(Native Method) 10.61.0.6: at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.6: at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.6: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.143: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.143: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.143:at java.security.AccessController.doPrivileged(Native Method) 10.61.0.143:at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.143:at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.143:at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.143: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.7: Exception in thread main java.lang.NoClassDefFoundError:
Re: Problem with building hadoop 0.21
Hey, Can you let us know why you want to replace all the jar files? That usually does not work, especially for development code in the code base. So, just use the one you have successfully compiled, don't replace jar files. Hope it can work. Simon 2011/2/27 朱韬 ryanzhu...@163.com Hi，guys: I checked out the source code fromhttp:// svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using this script: #!/bin/bash export JAVA_HOME=/usr/share/jdk1.6.0_14 export CFLAGS=-m64 export CXXFLAGS=-m64 export ANT_HOME=/opt/apache-ant-1.8.2 export PATH=$PATH:$ANT_HOME/bin ant -Dversion=0.21.0 -Dcompile.native=true -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar It was Ok before these steps. Then I replaced hadoop-mapred-0.21.0.jar, hadoop-mapred-0.21.0-sources.jar, hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files from the above step. Also I added my scheduler to lib. When starting the customed hadoop, I encountered the problems as blow: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/security/RefreshUserMappingsProtocol at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out 10.61.0.143: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out 10.61.0.7: starting tasktracker, logging to /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out 10.61.0.6: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.6: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.6: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.6: at java.security.AccessController.doPrivileged(Native Method) 10.61.0.6: at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.6: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.6: at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.6: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.143: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.143: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.143:at java.security.AccessController.doPrivileged(Native Method) 10.61.0.143:at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.143:at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.143:at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.143: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit. 10.61.0.7: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException 10.61.0.7: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException 10.61.0.7: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) 10.61.0.7: at java.security.AccessController.doPrivileged(Native Method) 10.61.0.7: at java.net.URLClassLoader.findClass(URLClassLoader.java:188) 10.61.0.7: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) 10.61.0.7: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 10.61.0.7: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) 10.61.0.7: at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) 10.61.0.7: Could not find the main class: org.apache.hadoop.mapred.TaskTracker. Program will exit.
Hadoop 0.21 HDFS fails to connect
Dear All,I have been trying to configure hadoop on a cluster but when i try to issue any comand regarding the hdfs like mkdir I get that the it is trying to connect to the server and after that it fails. I issued to commands the format and the mkdir but the fail please help and advise.RegardsAhmed ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ ./hadoop namenode -formatDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it. 11/02/04 12:04:03 INFO namenode.NameNode: STARTUP_MSG:/STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = cannonau.isti.cnr.it/146.48.82.190STARTUP_MSG: args = [-format]STARTUP_MSG: version = 0.21.0STARTUP_MSG: classpath = /home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/..:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-common-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-common-test-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0-sources.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-ant-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-test-0. A long list of paths STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010/11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.Re-format filesystem in /tmp/mylocal ? (Y or N) yFormat aborted in /tmp/mylocal11/02/04 12:04:14 INFO namenode.NameNode: SHUTDOWN_MSG:/SHUTDOWN_MSG: Shutting down NameNode at cannonau.isti.cnr.it/146.48.82.190/ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ /hadoop dfs -mkdir input-bash: /hadoop: No such file or directoryahmednagy@cannonau:~/HadoopStandalone/had oop-0.21.0/bin$ ./hadoop dfs -mkdir inputDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it. 11/02/04 12:04:30 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=3011/02/04 12:04:31 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id11/02/04 12:04:32 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 0 time(s).11/02/04 12:04:33 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 1 time(s).11/02/04 12:04:34 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 2 time(s).11/02/04 12:04:35 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 3 time(s).11/02/04 12:04:36 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 4 time(s).11/02/04 12:04:37 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. A lready tried 5 time(s).11/02/04 12:04:38 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 6 time(s).11/02/04 12:04:39 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 7 time(s).11/02/04 12:04:40 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 8 time(s).11/02/04 12:04:41 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 9 time(s).Bad connection to FS. command aborted.ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.-bash: Path: command not found 11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.Re-format filesystem in /tmp/mylocal ? (Y or N) yFormat aborted in /tmp/mylocal11/02/04 12:04:14 INFO namenode.NameNode: SHUTDOWN_MSG:/SHUTDOWN_MSG: Shutting down NameNode at
Re: installation of Hadoop 0.21
Thanks for your information. I was misled by the tutorial at http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. I can access the urls from NameNode - http://localhost:50070/ JobTracker - http://localhost:50030/ instead of from http://localhost:9100 and http://localhost:9101 mentioned in the tutorial. Jim On Tue, Jan 25, 2011 at 12:04 AM, li ping li.j...@gmail.com wrote: The exception java.io.IOException: NameNode is not formatted. indicated you should format the NameNode first. hadoop -fs namenode -format On Tue, Jan 25, 2011 at 12:47 PM, Jim X jim.p...@gmail.com wrote: I am trying to install Hadoop by following the instruction from http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. 1. I can not open http://localhost:9100 or http://localhost:9101 after I run bin/start-dfs.sh and bin/start-mapred.sh without any error message being printed. 2. I shutdown cygwin shell. 3. I start another cygwin shell, run bin/start-dfs.sh and get the following message from the shell. $ bin/start-dfs.sh starting namenode, logging to C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out localhost: datanode running as process 6908. Stop it first. localhost: secondarynamenode running as process 6156. Stop it first. Log message in C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed as below: 2011-01-24 23:10:11,202 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: 127.0.0.1/127.0.0.1:9100 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9100: readAndProcess threw exception java.io.IOException: Unable to read authentication method. Count of bytes read: 0 java.io.IOException: Unable to read authentication method at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-01-24 23:41:47,815 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-01-24 23:41:47,915 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) I am running Window Vista, JDK 1.6. I appreciate your help. Jim -- -李平
installation of Hadoop 0.21
I am trying to install Hadoop by following the instruction from http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. 1. I can not open http://localhost:9100 or http://localhost:9101 after I run bin/start-dfs.sh and bin/start-mapred.sh without any error message being printed. 2. I shutdown cygwin shell. 3. I start another cygwin shell, run bin/start-dfs.sh and get the following message from the shell. $ bin/start-dfs.sh starting namenode, logging to C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out localhost: datanode running as process 6908. Stop it first. localhost: secondarynamenode running as process 6156. Stop it first. Log message in C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed as below: 2011-01-24 23:10:11,202 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: 127.0.0.1/127.0.0.1:9100 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9100: readAndProcess threw exception java.io.IOException: Unable to read authentication method. Count of bytes read: 0 java.io.IOException: Unable to read authentication method at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-01-24 23:41:47,815 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-01-24 23:41:47,915 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) I am running Window Vista, JDK 1.6. I appreciate your help. Jim
Re: installation of Hadoop 0.21
The exception java.io.IOException: NameNode is not formatted. indicated you should format the NameNode first. hadoop -fs namenode -format On Tue, Jan 25, 2011 at 12:47 PM, Jim X jim.p...@gmail.com wrote: I am trying to install Hadoop by following the instruction from http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. 1. I can not open http://localhost:9100 or http://localhost:9101 after I run bin/start-dfs.sh and bin/start-mapred.sh without any error message being printed. 2. I shutdown cygwin shell. 3. I start another cygwin shell, run bin/start-dfs.sh and get the following message from the shell. $ bin/start-dfs.sh starting namenode, logging to C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out localhost: datanode running as process 6908. Stop it first. localhost: secondarynamenode running as process 6156. Stop it first. Log message in C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed as below: 2011-01-24 23:10:11,202 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: 127.0.0.1/127.0.0.1:9100 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9100: readAndProcess threw exception java.io.IOException: Unable to read authentication method. Count of bytes read: 0 java.io.IOException: Unable to read authentication method at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-01-24 23:41:47,815 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-01-24 23:41:47,915 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) I am running Window Vista, JDK 1.6. I appreciate your help. Jim -- -李平
Re: Caution using Hadoop 0.21
Two reasons - 1) we want a unit test to log whenever a write occurs 2) I want the keys generated by a write in a subsection of the app to be augmented by added data before being sent to hadoop On Mon, Nov 15, 2010 at 11:21 PM, Owen O'Malley omal...@apache.org wrote: I'm very sorry that you got burned by the change. Most MapReduce applications don't extend the Context classes since those are objects that are provided by the framework. In 0.21, we've marked which interfaces are stable and which are still evolving. We try and hold all of the interfaces stable, but evolving ones do change as we figure out what they should look like. Can I ask why you were extending the Context classes? -- Owen -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
Caution using Hadoop 0.21
Our group made a very poorly considered decision to build out cluster using Hadoop 0.21 We discovered that a number of programs written and running properly under 0.20.2 did not work under 0.21 The first issue is that Mapper.Context and Reducer.Context and many of their superclasses were converted from concrete classes to interfaces. This change, and I have never in 15 years of programming Java seen so major a change to well known public classes is guaranteed to break any code which subclasses these objects. While it is a far better decision to make these classes interface, the manner of the change and the fact that it is poorly documented shows extraordinary poor judgement on the part of the Hadoop developers http://lordjoesoftware.blogspot.com/ -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
Re: Caution using Hadoop 0.21
On Sat, Nov 13, 2010 at 4:33 PM, Shi Yu sh...@uchicago.edu wrote: I agree with Steve. That's why I am still using 0.19.2 in my production. Shi On 2010-11-13 12:36, Steve Lewis wrote: Our group made a very poorly considered decision to build out cluster using Hadoop 0.21 We discovered that a number of programs written and running properly under 0.20.2 did not work under 0.21 The first issue is that Mapper.Context and Reducer.Context and many of their superclasses were converted from concrete classes to interfaces. This change, and I have never in 15 years of programming Java seen so major a change to well known public classes is guaranteed to break any code which subclasses these objects. While it is a far better decision to make these classes interface, the manner of the change and the fact that it is poorly documented shows extraordinary poor judgement on the part of the Hadoop developers http://lordjoesoftware.blogspot.com/ At times we have been frustrated by rapidly changing API's # 23 August, 2010: release 0.21.0 available # 26 February, 2010: release 0.20.2 available # 14 September, 2009: release 0.20.1 available # 23 July, 2009: release 0.19.2 available # 22 April, 2009: release 0.20.0 available By the standard major/minor/revision scheme 0.20.X-0.21.X is a minor release. However since hadoop has never had a major release you might consider 0.20-0.21 to be a major release. In any case, are you saying that in 15 years of coding you have never seen an API change between minor releases? I think that is quite common. It was also more then a year between 0.20.X and 0.21.X. Again common to expect a change in that time frame.
Re: can not report progress from reducer context with hadoop 0.21
This is a bug in 0.21. MAPREDUCE-1905 (
https://issues.apache.org/jira/browse/MAPREDUCE-1905) is open for this.
On 9/21/10 4:29 PM, Marc Sturlese marc.sturl...@gmail.com wrote:
I am using hadoop 0.21
I have a reducer task wich takes more time to finish that the
mapreduce.task.timeout so it's being killed:
Task attempt_201009211103_0001_r_00_0 failed to report status for 602
seconds. Killing!
I have implemented a thread which is suposed to send progress and update the
status with an incremented counter but it seems not to be working. The
attempt is killed anyway.
I have tried an even simpler example: Not to use a thread and create an
infinite loop in the reducer which updates the status and sends progress on
each iteration... but the attempt keeps being killed:
@Override
public void reduce(Text keyName, IterableText paths, Context context)
throws IOException, InterruptedException {
while(true) {
context.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
context.setStatus(+context.getCounter(COUNTER_ADS.total_ads)) ;
context.progress();
}
context.write(new Text(done!), NullWritable.get());
}
I have even tryed to use TaskInputOutputContext insted of stright Context:
@Override
public void reduce(Text keyName, IterableText paths, Context context)
throws IOException, InterruptedException {
TaskInputOutputContext tac = (TaskInputOutputContext)context;
while(true) {
tac.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
tac.setStatus(+context.getCounter(COUNTER_ADS.total_ads)) ;
tac.progress();
}
context.write(new Text(done!), NullWritable.get());
}
Can anyone tell me what else could I try or what am I doing wrong? I am
really stuck on this problem and have no idea what else to do...
Thanks in advance
--
View this message in context:
http://lucene.472066.n3.nabble.com/can-not-report-progress-from-reducer-context-with-hadoop-0-21-tp1534700p1534700.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: can not report progress from reducer context with hadoop 0.21
Thanks, was going mad with this. It's working properly with 0.20.2 Once the patch is totally done will apply it to be able to keep using the MarkableIterator as it simplifies me many MapReduce jobs -- View this message in context: http://lucene.472066.n3.nabble.com/can-not-report-progress-from-reducer-context-with-hadoop-0-21-tp1534700p1555486.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: Hadoop 0.21 :: job.getCounters() returns null?
Hi Felix, Aaron Kimball hit the same problem - it's being discussed at https://issues.apache.org/jira/browse/MAPREDUCE-1920. Thanks for reporting this. Cheers, Tom On Tue, Jul 6, 2010 at 11:26 AM, Felix Halim felix.ha...@gmail.com wrote: I tried hadoop 0.21 release candidate. job.waitForCompletion(true); Counters ctrs = job.getCounters(); // here ctrs is null In the previous hadoop version 0.20.2 it worked fine for all times. Is this a bug in 0.21 ? Or i'm missing some settings? Thanks, Felix Halim
Hadoop 0.21 :: job.getCounters() returns null?
I tried hadoop 0.21 release candidate. job.waitForCompletion(true); Counters ctrs = job.getCounters(); // here ctrs is null In the previous hadoop version 0.20.2 it worked fine for all times. Is this a bug in 0.21 ? Or i'm missing some settings? Thanks, Felix Halim
Re: How to build and deploy Hadoop 0.21 ?
On Thu, Nov 5, 2009 at 2:34 AM, Andrei Dragomir adrag...@adobe.com wrote: Hello everyone. We ran into a bunch of issues with building and deploying hadoop 0.21. It would be great to get some answers about how things should work, so we can try to fix them. 1. When checking out the repositories, each of them can be built by itself perfectly. BUT, if you look in hdfs it has mapreduce libraries, and in mapreduce it has hdfs libraries. That's kind of a cross- reference between projects. Q: Is this dependence necessary ? Can we get rid of it ? Those are build-time dependencies. Ideally you'll ignore them post-build. Q: if it's necessary, how does one build the jars with the latest version of the source code ? how are the jars in the scm repository created (hadoop-hdfs/lib/hadoop-mapred-0.21-dev.jar) as long as there is a cross-reference ? 2. There are issues with the jar files and the webapps (dfshealth.jsp, etc). Right now, the only way to have a hadoop functioning system is to: build hdfs and mapreduce; copy everything from hdfs/build and mapreduce/build to common/build. Yup. Q: Is there a better way of doing this ? What needs to be fixed to have the webapps in the jar files (like on 0.20). Are there JIRA issues logged on this ? I have created a Makefile and some associated scripts that will build everything and squash it together for you; see https://issues.apache.org/jira/browse/HADOOP-6342 There is also a longer-term effort to use Maven to coordinate the three subprojects, and use a local repository for inter-project development on a single machine; see https://issues.apache.org/jira/browse/HADOOP-5107 for progress there. We would really appreciate some answers at least related to where hadoop is going with this build step, so we can help with patches / fixes. Thank you, Andrei Dragomirt
