On Thu, Apr 8, 2010 at 4:39 PM, Jean-Daniel Cryans <jdcry...@apache.org>wrote:
> Ted, please make sure you keep the conversation on the mailing list, > private answers have no value for an open source community. > > About JT, it would be awesome not to have to restart it but I don't > know enough about its internals to comment on how hard it would be to > modify. Maybe there's even an open jira. > > Or maybe someone else who listens could comment? Cloudera people? > > Rather than adding the jars to lib/ or HADOOP_CLASSPATH, you can also insert them directly into a lib/ directory inside your job jar. Then it will be added to the classpath of your MR Task child JVM when your job is localized on the task trackers. I think this is best practice for any jars that you anticipate will change on even a semiregular basis. -Todd > J-D > > On Thu, Apr 8, 2010 at 4:33 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > J-D: > > Do you think it makes for job tracker to dynamically refresh classpath ? > > I can see this need because HBase, HIVE, etc may initiate request in > cluster > > mode and we don't want to restart job tracker in production often. > > > > Cheers > > > > On Thu, Apr 8, 2010 at 3:52 PM, Jean-Daniel Cryans <jdcry...@apache.org> > > wrote: > >> > >> Actually the preferred method is putting the hbase and zookeeper jar > >> on the HADOOP_CLASSPATH of conf/hadoop-env.sh (putting the hbase conf > >> folder there is also good). > >> > >> Yeah have to restart JT, but AFAIK it's not HBase's fault right? ;) > >> > >> J-D > >> > >> On Thu, Apr 8, 2010 at 3:47 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> > I copied zookeeper-3.2.1.jar after seeing the exception below. > >> > Turns out that I have to restart job tracker to refresh classpath :-( > >> > > >> > On Thu, Apr 8, 2010 at 3:03 PM, Jean-Daniel Cryans > >> > <jdcry...@apache.org>wrote: > >> > > >> >> Zookeeper jar isn't in the path of hadoop? > >> >> > >> >> btw https://issues.apache.org/jira/browse/HBASE-2335 > >> >> > >> >> J-D > >> >> > >> >> On Thu, Apr 8, 2010 at 2:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> >> > I ran hbase export in cluster mode. > >> >> > > >> >> > From Saturn: > >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: zookeeper. > >> >> > disableAutoWatchReset is false > >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Attempting connection > to > >> >> server > >> >> > uranus/10.10.31.18:2181 > >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Priming connection to > >> >> > java.nio.channels.SocketChannel[connected local=/10.10.31.16:34170 > >> >> remote=uranus/ > >> >> > 10.10.31.18:2181] > >> >> > 10/04/07 14:18:22 INFO zookeeper.ClientCnxn: Server connection > >> >> > successful > >> >> > 10/04/07 14:18:22 INFO mapreduce.TableInputFormatBase: split: > >> >> 0*->Neptune*:, > >> >> > 10/04/07 14:18:23 INFO mapred.JobClient: Running job: > >> >> job_201004051529_0015 > >> >> > 10/04/07 14:18:24 INFO mapred.JobClient: map 0% reduce 0% > >> >> > 10/04/07 14:18:36 INFO mapred.JobClient: Task Id : > >> >> > attempt_201004051529_0015_m_000000_0, Status : FAILED > >> >> > Error: java.lang.ClassNotFoundException: > org.apache.zookeeper.Watcher > >> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >> >> > at java.security.AccessController.doPrivileged(Native > Method) > >> >> > at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> >> > at > >> >> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> >> > at > >> >> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > >> >> > at java.lang.ClassLoader.defineClass1(Native Method) > >> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:621) > >> >> > at > >> >> > > >> >> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) > >> >> > at > >> >> > java.net.URLClassLoader.defineClass(URLClassLoader.java:260) > >> >> > at > java.net.URLClassLoader.access$000(URLClassLoader.java:56) > >> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > >> >> > at java.security.AccessController.doPrivileged(Native > Method) > >> >> > at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> >> > at > >> >> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> >> > at > >> >> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager.getClientZooKeeperWatcher(HConnectionManager.java:151) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getZooKeeperWrapper(HConnectionManager.java:885) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:901) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:580) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:630) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:589) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:630) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:593) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:556) > >> >> > at > >> >> > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:127) > >> >> > at > >> >> > org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:105) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:73) > >> >> > at > >> >> > > >> >> > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > >> >> > at > >> >> > > >> >> > >> >> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > >> >> > at > >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:536) > >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170) > >> >> > > >> >> > I wonder why the above exception was thrown because we have same > >> >> > hbase > >> >> > deployment on all the servers: > >> >> > [r...@neptune software]# ls -l lib/zookeeper-3.2.1.jar > >> >> > -rwxr-xr-x 1 hbaseadmin hbaseadmin 913093 Oct 7 11:55 > >> >> > lib/zookeeper-3.2.1.jar > >> >> > > >> >> > Can someone shed some light on how to get over the above exception > ? > >> >> > > >> >> > Thanks > >> >> > > >> >> > >> > > > > > > -- Todd Lipcon Software Engineer, Cloudera
