Hi Hari, getConf() returns a Hadoop configuration instance, that does not include HBase's. You need to call
Configuration conf = HBaseConfiguration.create(); Job job = new Job(conf); for it to be read. Lars On Mon, Nov 22, 2010 at 11:32 AM, Hari Sreekumar <[email protected]> wrote: > Hi, > I tried using this code to read configuration (I am using Toolrunner) : > > public int run(String[] args) throws Exception { > String fileName = args[0]; > Path inputPath = new Path(args[0]); > > Job job = new Job(getConf()); > System.out.println("Conf_hbase: " + > getConf().get("hbase.zookeeper.quorum")); > System.out.println("Conf_hadoop: " + > getConf().get("dfs.replication")); > > job.setJarByClass(BulkUpload.class); > FileInputFormat.setInputPaths(job, inputPath); > job.setJobName(NAME + "_" + fileName); > > job.setMapperClass(BulkUploadMapper.class); > job.setInputFormatClass(TextInputFormat.class); > > TableMapReduceUtil.initTableReducerJob("CustomerData", null, job); > //System.out.println("") > job.setNumReduceTasks(0); > > boolean success = job.waitForCompletion(true); > return success ? 0 : 1; > } > > Is this code right to get the configuration? Because I am getting both > dfs.replication and hbase.zookeeper.quorum as null this way. But when I > change dfs.replication in my config file, I do see a change in replication > after upload into HDFS. > > On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar > <[email protected]>wrote: > >> Hi Lars, >> >> I start them through HBase implicitly. I'll try printing the config >> values and post. >> >> thanks, >> Hari >> >> >> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <[email protected]>wrote: >> >>> Hi Hari, >>> >>> Are you starting them yourself or have HBase start them for you >>> implicitly? >>> >>> Lars >>> >>> On Nov 22, 2010, at 6:19, Hari Sreekumar <[email protected]> >>> wrote: >>> >>> > Hey Lars, >>> > >>> > I have HQuorumPeer running on all nodes that I specify in my >>> > hbase-site file. One thing I wanted to clarify.. what is the default >>> value >>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my >>> > hbase-env.sh file. >>> > >>> > thanks, >>> > hari >>> > >>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <[email protected]> >>> wrote: >>> > >>> >> Hi Hari, >>> >> >>> >> On which of these for machines do you have a ZooKeeper daemon running >>> as >>> >> well? >>> >> >>> >> Lars >>> >> >>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar >>> >> <[email protected]> wrote: >>> >>> Hi, >>> >>> >>> >>> But it is reading settings from hbase-site.xml. If it was not >>> reading >>> >> my >>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber to >>> the >>> >>> quroum right? After all, it is responding to changes I make in my xml >>> >> file. >>> >>> What else can be the issue here? >>> >>> >>> >>> hari >>> >>> >>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <[email protected]> >>> >> wrote: >>> >>> >>> >>>> Hi Hari, >>> >>>> >>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is >>> >> missing >>> >>>> from the classpath on the clients. Did you pack it into the jar? >>> >>>> >>> >>>> And yes, even one ZK server is fine in such a small cluster. >>> >>>> >>> >>>> You can see it is trying to connect to localhost which is the default >>> if >>> >>>> the site file is missing. >>> >>>> >>> >>>> Regards, >>> >>>> Lars >>> >>>> >>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <[email protected]> >>> >>>> wrote: >>> >>>> >>> >>>>> Hi, >>> >>>>> Is it necessary that all RegionServers must also be part of the ZK >>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and >>> >>>> hadoop2, >>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for >>> >>>> HBase). >>> >>>>> >>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property: >>> >>>>> <name>hbase.zookeeper.quorum</name> >>> >>>>> <value>hadoop1,hadoop2,hadoop3</value> >>> >>>>> >>> >>>>> I get this exception for all tasks that run on ejabber(the 4th >>> node): >>> >>>>> >>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn: >>> >> Attempting >>> >>>>> connection to server localhost/127.0.0.1:2181 >>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn: >>> >> Exception >>> >>>>> closing session 0x0 to sun.nio.ch.selectionkeyi...@7c2e1f1f >>> >>>>> java.net.ConnectException: Connection refused >>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> >>>>> at >>> >>>>> >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >>>>> at >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: >>> Ignoring >>> >>>>> exception during shutdown input >>> >>>>> java.nio.channels.ClosedChannelException >>> >>>>> at >>> >>>>> >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>> >>>>> at >>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>> >>>>> at >>> >>>>> >>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) >>> >>>>> at >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) >>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn: >>> Ignoring >>> >>>>> exception during shutdown output >>> >>>>> java.nio.channels.ClosedChannelException >>> >>>>> at >>> >>>>> >>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >>> >>>>> at >>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >>> >>>>> at >>> >>>>> >>> >> >>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) >>> >>>>> at >>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) >>> >>>>> 2010-11-21 23:35:47,925 WARN >>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create >>> >>>> /hbase >>> >>>>> -- check quorum servers, currently=localhost:2181 >>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase >>> >>>>> at >>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) >>> >>>>> at >>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) >>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) >>> >>>>> at >>> >>>>> >>> >>>> >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) >>> >>>>> at >>> >>>>> >>> >>>> >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) >>> >>>>> at >>> >>>>> >>> >>>> >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) >>> >>>>> >>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't >>> >> get >>> >>>> this >>> >>>>> exception. My understanding was that a small cluster like mine >>> should >>> >>>> only >>> >>>>> need one ZK machine. >>> >>>>> >>> >>>>> Thanks, >>> >>>>> Hari >>> >>>> >>> >>> >>> >> >>> >> >> >
