Hi,
I tried using this code to read configuration (I am using Toolrunner) :
public int run(String[] args) throws Exception {
String fileName = args[0];
Path inputPath = new Path(args[0]);
Job job = new Job(getConf());
System.out.println("Conf_hbase: " +
getConf().get("hbase.zookeeper.quorum"));
System.out.println("Conf_hadoop: " +
getConf().get("dfs.replication"));
job.setJarByClass(BulkUpload.class);
FileInputFormat.setInputPaths(job, inputPath);
job.setJobName(NAME + "_" + fileName);
job.setMapperClass(BulkUploadMapper.class);
job.setInputFormatClass(TextInputFormat.class);
TableMapReduceUtil.initTableReducerJob("CustomerData", null, job);
//System.out.println("")
job.setNumReduceTasks(0);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
Is this code right to get the configuration? Because I am getting both
dfs.replication and hbase.zookeeper.quorum as null this way. But when I
change dfs.replication in my config file, I do see a change in replication
after upload into HDFS.
On Mon, Nov 22, 2010 at 2:36 PM, Hari Sreekumar <[email protected]>wrote:
> Hi Lars,
>
> I start them through HBase implicitly. I'll try printing the config
> values and post.
>
> thanks,
> Hari
>
>
> On Mon, Nov 22, 2010 at 1:58 PM, Lars George <[email protected]>wrote:
>
>> Hi Hari,
>>
>> Are you starting them yourself or have HBase start them for you
>> implicitly?
>>
>> Lars
>>
>> On Nov 22, 2010, at 6:19, Hari Sreekumar <[email protected]>
>> wrote:
>>
>> > Hey Lars,
>> >
>> > I have HQuorumPeer running on all nodes that I specify in my
>> > hbase-site file. One thing I wanted to clarify.. what is the default
>> value
>> > of HBASE_MANAGES_ZK ? Because I have not explicitly set it to true in my
>> > hbase-env.sh file.
>> >
>> > thanks,
>> > hari
>> >
>> > On Mon, Nov 22, 2010 at 10:39 AM, Lars George <[email protected]>
>> wrote:
>> >
>> >> Hi Hari,
>> >>
>> >> On which of these for machines do you have a ZooKeeper daemon running
>> as
>> >> well?
>> >>
>> >> Lars
>> >>
>> >> On Mon, Nov 22, 2010 at 5:51 AM, Hari Sreekumar
>> >> <[email protected]> wrote:
>> >>> Hi,
>> >>>
>> >>> But it is reading settings from hbase-site.xml. If it was not
>> reading
>> >> my
>> >>> changes, the problem wouldn't have gotten fixed when I add ejabber to
>> the
>> >>> quroum right? After all, it is responding to changes I make in my xml
>> >> file.
>> >>> What else can be the issue here?
>> >>>
>> >>> hari
>> >>>
>> >>> On Mon, Nov 22, 2010 at 12:54 AM, Lars George <[email protected]>
>> >> wrote:
>> >>>
>> >>>> Hi Hari,
>> >>>>
>> >>>> You are missing the quorum setting. It seems the hbase-site.xml is
>> >> missing
>> >>>> from the classpath on the clients. Did you pack it into the jar?
>> >>>>
>> >>>> And yes, even one ZK server is fine in such a small cluster.
>> >>>>
>> >>>> You can see it is trying to connect to localhost which is the default
>> if
>> >>>> the site file is missing.
>> >>>>
>> >>>> Regards,
>> >>>> Lars
>> >>>>
>> >>>> On Nov 21, 2010, at 19:22, Hari Sreekumar <[email protected]>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>> Is it necessary that all RegionServers must also be part of the ZK
>> >>>>> Quorum? I have a 4 node cluster, with node hadoop1 being master and
>> >>>> hadoop2,
>> >>>>> hadoop3 and ejabber being the slaves (Both in case of hadoop and for
>> >>>> HBase).
>> >>>>>
>> >>>>> When I keep only 3 nodes in the zookeeper.quorum property:
>> >>>>> <name>hbase.zookeeper.quorum</name>
>> >>>>> <value>hadoop1,hadoop2,hadoop3</value>
>> >>>>>
>> >>>>> I get this exception for all tasks that run on ejabber(the 4th
>> node):
>> >>>>>
>> >>>>> 2010-11-21 23:35:47,785 INFO org.apache.zookeeper.ClientCnxn:
>> >> Attempting
>> >>>>> connection to server localhost/127.0.0.1:2181
>> >>>>> 2010-11-21 23:35:47,790 WARN org.apache.zookeeper.ClientCnxn:
>> >> Exception
>> >>>>> closing session 0x0 to sun.nio.ch.selectionkeyi...@7c2e1f1f
>> >>>>> java.net.ConnectException: Connection refused
>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>>> at
>> >>>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> Ignoring
>> >>>>> exception during shutdown input
>> >>>>> java.nio.channels.ClosedChannelException
>> >>>>> at
>> >>>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> >>>>> at
>> >> sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> >>>>> at
>> >>>>>
>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>>>> 2010-11-21 23:35:47,791 WARN org.apache.zookeeper.ClientCnxn:
>> Ignoring
>> >>>>> exception during shutdown output
>> >>>>> java.nio.channels.ClosedChannelException
>> >>>>> at
>> >>>>>
>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> >>>>> at
>> >> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> >>>>> at
>> >>>>>
>> >>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> >>>>> at
>> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> >>>>> 2010-11-21 23:35:47,925 WARN
>> >>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create
>> >>>> /hbase
>> >>>>> -- check quorum servers, currently=localhost:2181
>> >>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>>> KeeperErrorCode = ConnectionLoss for /hbase
>> >>>>> at
>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> >>>>> at
>> >>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>> >>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
>> >>>>> at
>> >>>>>
>> >>>>
>> >>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
>> >>>>>
>> >>>>> When I add ejabber also to the ZK quorum and restart HBase, I don't
>> >> get
>> >>>> this
>> >>>>> exception. My understanding was that a small cluster like mine
>> should
>> >>>> only
>> >>>>> need one ZK machine.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Hari
>> >>>>
>> >>>
>> >>
>>
>
>
