Hi, comments inline:
On 16.08.2011 06:42, Stack wrote:
Oh, are you running on hbase trunk? If so, did you write all your
data with TRUNK or did you start up 0.92 on a data that was written w/
0.90 (It should work but you may have hit an issue).
I had branch-0.90 and after proper shutdown i had updated hbase to
trunk, it looks like -ROOT- table was migrated but .META. had problem
with saving information about migration in logs i had informations:
2011-08-18 17:16:09,334 INFO org.apache.hadoop.hbase.master.HMaster:
.META. assigned=2, rit=false, location=lala.golden,60020,1313680556932
2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster:
metaUpdated = NULL.
2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster:
Meta updated status = false
2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster:
Meta has HRI with HTDs. Updating meta now.
2011-08-18 17:16:17,535 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Updated -ROOT- row with metaMigrated status = true
2011-08-18 17:16:17,536 INFO org.apache.hadoop.hbase.master.HMaster:
Meta updated with new HRI.
and even after restarting i saw this in logs.
./hbase shell show all one regions, but any operation on tables show
information that .META. table doesn't exist.
Full logs from first and second run after upgrade are here:
http://s2.e-back.net/hbase_logs.tar.gz
-ROOT- have this rows:
hbase(main):005:0> scan '-ROOT-'
ROW
COLUMN+CELL
-ROOT-,,0
column=info:metamigrated, timestamp=1313680577531, value=\xFF
.META.,,1
column=info:regioninfo, timestamp=1278560826938, value=REGION => {NAME
=> '.META.,,1 TableName => ', STARTKEY => '', ENDKEY => '', ENCODED =>
1028785192,}
.META.,,1
column=info:server, timestamp=1313680569288, value=lala.golden:60020
.META.,,1
column=info:serverstartcode, timestamp=1313680569288, value=1313680556932
2 row(s) in 0.0500 seconds
On 16.08.2011 06:40, Stack wrote:
What version of hbase is this Sebastian?
What you mean by remove -ROOT- and .META. (that doesn't sound good but
you probably mean something else though looking at the exception
below, maybe this is what you did).
Long time ago when i smashed my hbase instalation when i removed .META.
and -ROOT- (./bin/hadoop fs -rmr /hbase/.META.) and run add_table.rb
script all tables were online again.
St.Ack
On Thu, Aug 11, 2011 at 11:54 AM, Sebastian Bauer<[email protected]> wrote:
2011-08-11 20:53:55,889 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e.
to db2a.goldenline.pl,60020,1313088827048, trying to assign elsewhere
instead; retry=
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845)
at
org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818)
at
org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
at $Proxy7.openRegion(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101)
at
org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296)
at
org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300)
2011-08-11 20:53:55,895 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e.
to db2b.goldenline.pl,60020,1313088826553, trying to assign elsewhere
instead; retry=
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845)
at
org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818)
at
org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
at $Proxy7.openRegion(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101)
at
org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355)
at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296)
at
org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300)
--
Pozdrawiam
Sebastian Bauer
-----------------------------------------------------
http://tikecik.pl
