[
https://issues.apache.org/jira/browse/HBASE-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622826#action_12622826
]
stack commented on HBASE-826:
-----------------------------
More testing would seem to indicate the above described fix works for at least
the small testcase where small case was upload of 2M rows into a table, delete
it, then try same upload. Was failing reliably before this fix.
Trying with a bigger table.
Debugging this issue, I made getStores public in HRegion and then in HStores, I
made getReader also public. Doing this, I was able to do a little script that
could enumerate all keys in the .META. table in a few lines. I think we should
make these methods public to facilitate future easier debugging.
Script looked like this:
{code}
# Script that outputs all keys in .META. store files.
# HBase MUST be shutdown when you run this script.
#
# To run it, do: ${HBASE_HOME}/bin/hbase org.jruby.Main iteratemeta.rb
include Java
import org.apache.hadoop.hbase.util.MetaUtils
import org.apache.hadoop.hbase.HRegionInfo
import org.apache.hadoop.hbase.HConstants
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.HStoreKey
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.regionserver.HLogEdit
# Get configuration to use.
c = HBaseConfiguration.new()
# Set hadoop filesystem configuration using the hbase.rootdir.
# Otherwise, we'll always use localhost though the hbase.rootdir
# might be pointing at hdfs location.
c.set("fs.default.name", c.get(HConstants::HBASE_DIR));
meta = utils.getMetaRegion(HRegionInfo::FIRST_META_REGIONINFO)
begin
for columnDescriptor in meta.getRegionInfo().getTableDesc().getFamilies()
store = meta.getStore(columnDescriptor.getName())
puts "Store: %s" % [store.toString()]
for reader in store.getReaders()
puts "Store: %s, reader %s" % [store.toString(), reader.toString()]
key = HStoreKey.new()
value = ImmutableBytesWritable.new()
while (reader.next(key, value))
puts "%s %s" % [key, (HLogEdit.isDeleted(value.get()) ? "d": "")]
end
end
end
ensure
utils.shutdown()
end
{code}
> delete table followed by recreation results in honked table
> -----------------------------------------------------------
>
> Key: HBASE-826
> URL: https://issues.apache.org/jira/browse/HBASE-826
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.2.1, 0.3.0
>
>
> Daniel Leffel suspected that delete and then recreate causes issues. I tried
> it on our little cluster. I'm doing a MR load up into the newly created
> table and after a few million rows, the MR job just hangs. Its looking for a
> region that doesn't exist:
> {code}
> 2008-08-13 03:32:36,840 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2008-08-13 03:32:36,940 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 1
> 2008-08-13 03:32:37,420 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
> REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
> FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
> VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}]}}
> 2008-08-13 03:32:37,541 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading
> table servers because: HRegionInfo was null or empty in .META.
> 2008-08-13 03:32:37,541 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed
> .META.,,1 from cache because of TestTable,0008388608,99999999999999
> 2008-08-13 03:32:37,544 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
> REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED =>
> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
> FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE',
> VERSIONS => '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
> BLOCKCACHE => 'false'}]}}
> 2008-08-13 03:32:47,605 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading
> table servers because: HRegionInfo was null or empty in .META.
> 2008-08-13 03:32:47,606 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed
> .META.,,1 from cache because of TestTable,0008388608,99999999999999
> ....
> {code}
> My guess is that its a region that was in the tables' previous incarnation
> with ghosts left over down inside .META.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
