Thank a lot for your suggesstions Ram. I will see the code. Meanwhile I just
now evaluated with Hbase 0.90.4 and it works fine after throwing few
exceptions - Socket connections with old master then,
2011-11-03 02:14:49,578 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: RemoteException
connecting to master
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:734)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:595)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.ServerManager.raceThatShouldNotHappenAnymore(ServerManager.java:307)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:273)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:638)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.regionServerReport(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:728)
... 2 more
After this exception, I got messages like,
Attempting to connect to Master at M4 and then,
Connected to master at M4.
Ramkrishna S Vasudevan wrote:
>
>
>
> If you still cannot move out of 0.90.0 try to understand the code in
> HRegionServer.reportForDuty() in 0.90.3 and merge it in your copy of
> 0.90.0
> and try. Just a suggestion :)
>
> Regards
> Ram
>
>
> -----Original Message-----
> From: V.Sriram [mailto:[email protected]]
> Sent: Thursday, November 03, 2011 1:11 PM
> To: [email protected]
> Subject: RE: RE: Hbase Multiple Master issue
>
>
> Thats great news. Thanks for ur valuable info!! I will use that =)
>
> Stuti Awasthi wrote:
>>
>> Hi Sriram,
>> I am using Hbase 0.90.3 and I also tested Multiple master issue. It
>> worked fine for my cluster. Once new Master is up, after sometime
>> region server start communicating with new Master.
>> Hope this helps.
>>
>> -----Original Message-----
>> From: V.Sriram [mailto:[email protected]]
>> Sent: Thursday, November 03, 2011 12:54 PM
>> To: [email protected]
>> Subject: Re: RE: Hbase Multiple Master issue
>>
>>
>> I saw the release notes of Hbase 0.90.2, 0.90.3 and 0.90.4. This issue
>> doesn't seem to be mentioned there. Also since our team want continue
>> with 0.90.0 itself for some more time, I am not currently upgrading to
>> newer versions.
>>
>> Today saw the Zk_dump to verify if the new master is currently
>> reflected in Zookeeper. It is correctly reflected. But even then the
>> region servers don't communicate with the new master.
>>
>> I think region servers would ideally communicate with zookeeper and
>> get the new master node and then communicate with it. But thats not
>> happenning!!!
>> Any suggestions plz !!
>>
>>
>> V.Sriram wrote:
>>>
>>> Thanks for the reply. Will try other versions as well!!
>>>
>>> regards,
>>> V.Sriram
>>>
>>>
>>> Ramkrishna S Vasudevan wrote:
>>>>
>>>> Hi
>>>>
>>>> If i remember correctly this issue was fixed in later versions of
>>>> 0.90.0.
>>>> Could you try this in other versions and see if this problem
>>>> persists..ideally it should not.
>>>>
>>>>
>>>> Regards
>>>> Ram
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "V.Sriram" <[email protected]>
>>>> Date: Tuesday, November 1, 2011 3:01 pm
>>>> Subject: RE: Hbase Multiple Master issue
>>>> To: [email protected]
>>>>
>>>>>
>>>>> I am using 0.90.0 version
>>>>>
>>>>> regards,
>>>>> V.Sriram
>>>>>
>>>>>
>>>>> Ramkrishna S Vasudevan wrote:
>>>>> >
>>>>> > Hi
>>>>> >
>>>>> > What version of HBase are you using?
>>>>> >
>>>>> > Regards
>>>>> > Ram
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: V.Sriram [mailto:[email protected]]
>>>>> > Sent: Tuesday, November 01, 2011 12:58 PM
>>>>> > To: [email protected]
>>>>> > Subject: Hbase Multiple Master issue
>>>>> >
>>>>> >
>>>>> > Hi all,
>>>>> >
>>>>> > I am evaluating HBase Multiple master utility as I require that
>>>>> > funtionality. I used a simple 4 machine hbase cluster with M1 as
>>>>> master.> Now
>>>>> > I started the Hmaster in M4 as well. I could see logs in M4
>>>>> Master log as,
>>>>> > This HMaster is waiting as already M1 is acting as HMaster.
>>>>> >
>>>>> > After this I stopped the Master in the M1 and after few seconds,
>>>>> the M4
>>>>> > master took over. But I am not seeing any region server
>>>>> communicating to
>>>>> > this new master. The region server logs show,
>>>>> >
>>>>> > 2011-10-31 23:21:18,834 WARN
>>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to
>>>>> connect to
>>>>> > master. Retrying. Error was:
>>>>> > java.net.ConnectException: Connection refused
>>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> > at
>>>>> >
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:5
>>>>> 74)>
>>>>> at
>>>>> >
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeo
>>>>> u
>>>>> t.java:2>
>>>>> 06)
>>>>> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(H
>>>>> B
>>>>> aseClien>
>>>>> t.java:311)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.j
>>>>> ava:865)>
>>>>> at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:2
>>>>> 57)> at $Proxy3.getProtocolVersion(Unknown Source)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>>>>> > at
>>>>> > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegio
>>>>> n
>>>>> Server.j>
>>>>> ava:1443)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerR
>>>>> e
>>>>> port(HRe>
>>>>> gionServer.java:737)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServe
>>>>> r
>>>>> .java:58>
>>>>> 6)
>>>>> > at java.lang.Thread.run(Thread.java:619)
>>>>> >
>>>>> > Also in the M4 Master log I am seeing,
>>>>> >
>>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>>> > listener.getLocalPort() returned 60010
>>>>> > webServer.getConnectors()[0].getLocalPort() returned 60010
>>>>> > 2011-10-31 23:20:47,671 INFO org.apache.hadoop.http.HttpServer:
>>>>> Jetty> bound
>>>>> > to port 60010
>>>>> > 2011-10-31 23:20:47,671 INFO org.mortbay.log: jetty-6.1.26
>>>>> > 2011-10-31 23:20:48,691 INFO org.mortbay.log: Started
>>>>> > [email protected]:60010
>>>>> > 2011-10-31 23:20:48,691 DEBUG
>>>>> org.apache.hadoop.hbase.master.HMaster:> Started service threads
>>>>> > 2011-10-31 23:20:50,192 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> > 2011-10-31 23:20:51,695 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> > 2011-10-31 23:20:53,197 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:> Waiting on
>>>>> regionserver(s) to checkin
>>>>> >
>>>>> > Seems the region servers are only communicating to the M1 Hmaster
>>>>> and not
>>>>> > to
>>>>> > the new one!!
>>>>> >
>>>>> > Any help regarding this is highly appreciated :handshake:
>>>>> > --
>>>>> > View this message in context:
>>>>> > http://old.nabble.com/Hbase-Multiple-Master-issue-
>>>>> tp32757197p32757197.html> Sent from the HBase User mailing list
>>>>> archive at Nabble.com.
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>> --
>>>>> View this message in context: http://old.nabble.com/Hbase-Multiple-
>>>>> Master-issue-tp32757197p32757528.html
>>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770898.
>> html Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>> ::DISCLAIMER::
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> It shall not attach any liability on the originator or HCL or its
>> affiliates. Any views or opinions presented in this email are solely
>> those of the author and may not necessarily reflect the opinions of
>> HCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure,
>> modification, distribution and / or publication of this message
>> without the prior written consent of the author of this e-mail is
>> strictly prohibited. If you have received this email in error please
>> delete it and notify the sender immediately. Before opening any mail
>> and attachments please check them for viruses and defect.
>>
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32770950.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
>
--
View this message in context:
http://old.nabble.com/Hbase-Multiple-Master-issue-tp32757197p32771444.html
Sent from the HBase User mailing list archive at Nabble.com.
