Can you redo with logs at DEBUG level? This seems like a migration issue -- i.e. migrating the catalog tables from old format to new -- but from what I can see in the logs, its more an issue w/ state transitions in zk... I need the DEBUG logging to tell for sure.
St.Ack On Thu, Aug 18, 2011 at 8:27 AM, Sebastian Bauer <[email protected]> wrote: > Hi, comments inline: > > On 16.08.2011 06:42, Stack wrote: >> >> Oh, are you running on hbase trunk? If so, did you write all your >> data with TRUNK or did you start up 0.92 on a data that was written w/ >> 0.90 (It should work but you may have hit an issue). > > I had branch-0.90 and after proper shutdown i had updated hbase to trunk, it > looks like -ROOT- table was migrated but .META. had problem with saving > information about migration in logs i had informations: > > 2011-08-18 17:16:09,334 INFO org.apache.hadoop.hbase.master.HMaster: .META. > assigned=2, rit=false, location=lala.golden,60020,1313680556932 > 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster: > metaUpdated = NULL. > 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster: Meta > updated status = false > 2011-08-18 17:16:09,340 INFO org.apache.hadoop.hbase.master.HMaster: Meta > has HRI with HTDs. Updating meta now. > 2011-08-18 17:16:17,535 INFO org.apache.hadoop.hbase.catalog.MetaEditor: > Updated -ROOT- row with metaMigrated status = true > 2011-08-18 17:16:17,536 INFO org.apache.hadoop.hbase.master.HMaster: Meta > updated with new HRI. > > and even after restarting i saw this in logs. > > ./hbase shell show all one regions, but any operation on tables show > information that .META. table doesn't exist. > Full logs from first and second run after upgrade are here: > http://s2.e-back.net/hbase_logs.tar.gz > > -ROOT- have this rows: > hbase(main):005:0> scan '-ROOT-' > ROW > COLUMN+CELL > -ROOT-,,0 > column=info:metamigrated, timestamp=1313680577531, value=\xFF > .META.,,1 > column=info:regioninfo, timestamp=1278560826938, value=REGION => {NAME => > '.META.,,1 TableName => ', STARTKEY => '', ENDKEY => '', ENCODED => > 1028785192,} > .META.,,1 > column=info:server, timestamp=1313680569288, value=lala.golden:60020 > .META.,,1 > column=info:serverstartcode, timestamp=1313680569288, value=1313680556932 > 2 row(s) in 0.0500 seconds > > > On 16.08.2011 06:40, Stack wrote: >> >> What version of hbase is this Sebastian? >> >> What you mean by remove -ROOT- and .META. (that doesn't sound good but >> you probably mean something else though looking at the exception >> below, maybe this is what you did). > > Long time ago when i smashed my hbase instalation when i removed .META. and > -ROOT- (./bin/hadoop fs -rmr /hbase/.META.) and run add_table.rb script all > tables were online again. >> >> St.Ack >> >> On Thu, Aug 11, 2011 at 11:54 AM, Sebastian Bauer<[email protected]> >> wrote: >>> >>> 2011-08-11 20:53:55,889 WARN >>> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of >>> >>> Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e. >>> to db2a.goldenline.pl,60020,1313088827048, trying to assign elsewhere >>> instead; retry= >>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: >>> java.lang.IllegalArgumentException: Can not create a Path from an empty >>> string >>> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:90) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:50) >>> at >>> org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845) >>> at >>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818) >>> at >>> >>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832) >>> at >>> >>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127) >>> at >>> >>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99) >>> at >>> >>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> >>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312) >>> at >>> >>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222) >>> >>> at >>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871) >>> at >>> >>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141) >>> at $Proxy7.openRegion(Unknown Source) >>> at >>> >>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014) >>> at >>> >>> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243) >>> at >>> >>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467) >>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300) >>> >>> 2011-08-11 20:53:55,895 WARN >>> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of >>> >>> Golden_ATU,m_ef06511a814155022686d50cebb0dba5_2010_9_40CC7C8170915ABC43432E524F0E6D2B,1301066006080.f656ca602b8638ad8329c36af85ae43e. >>> to db2b.goldenline.pl,60020,1313088826553, trying to assign elsewhere >>> instead; retry= >>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: >>> java.lang.IllegalArgumentException: Can not create a Path from an empty >>> string >>> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:90) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:50) >>> at >>> org.apache.hadoop.hbase.util.FSUtils.getTablePath(FSUtils.java:845) >>> at >>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoPath(FSUtils.java:818) >>> at >>> >>> org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:832) >>> at >>> >>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127) >>> at >>> >>> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99) >>> at >>> >>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2329) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> >>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312) >>> at >>> >>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1222) >>> >>> at >>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:871) >>> at >>> >>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141) >>> at $Proxy7.openRegion(Unknown Source) >>> at >>> >>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:404) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1239) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1034) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1014) >>> at >>> >>> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:101) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:458) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:389) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:355) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:296) >>> at >>> >>> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:243) >>> at >>> >>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:467) >>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:300) >>> >>> -- >>> >>> Pozdrawiam >>> Sebastian Bauer >>> ----------------------------------------------------- >>> http://tikecik.pl >>> >>> >
