[
https://issues.apache.org/jira/browse/HBASE-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017212#comment-13017212
]
stack commented on HBASE-3744:
------------------------------
Here is the exception I ran into:
{code}
2011-04-07 14:48:26,788 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs --
HDFS-200
2011-04-07 14:48:26,789 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: New
hlog
/hbase/jeichorn_test/710a5516e0de5da60ed7ce9466ac3b6f/.logs/hlog.1302212906763
2011-04-07 14:48:26,789 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Using getNumCurrentReplicas--HDFS-826
2011-04-07 14:48:26,789 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.
2011-04-07 14:48:26,805 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.; next
sequenceid=1
2011-04-07 14:48:26,807 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Added
region jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f. to META
2011-04-07 14:48:26,807 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.:
disabling compactions & flushes
2011-04-07 14:48:26,807 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region
jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.
2011-04-07 14:48:26,807 DEBUG org.apache.hadoop.hbase.regionserver.Store:
closed default
2011-04-07 14:48:26,807 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.
2011-04-07 14:48:26,807 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
IPC Server handler 5 on 60000.logSyncer interrupted while waiting for sync
requests
2011-04-07 14:48:26,807 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC
Server handler 5 on 60000.logSyncer exiting
2011-04-07 14:48:26,808 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
closing hlog writer in
hdfs://sv2borg169:9000/hbase/jeichorn_test/710a5516e0de5da60ed7ce9466ac3b6f/.logs
2011-04-07 14:48:26,816 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Moved 1 log files to
/hbase/jeichorn_test/710a5516e0de5da60ed7ce9466ac3b6f/.oldlogs
2011-04-07 14:48:26,817 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 1 region(s) round-robin across 9 server(s)
2011-04-07 14:48:26,817 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg160,60020,1302129205464
2011-04-07 14:48:26,818 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg161,60020,1302129205463
2011-04-07 14:48:26,818 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg160,60020,1302129205464
2011-04-07 14:48:26,818 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg162,60020,1302129205472
2011-04-07 14:48:26,819 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg163,60020,1302129205458
2011-04-07 14:48:26,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg161,60020,1302129205463
2011-04-07 14:48:26,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg162,60020,1302129205472
2011-04-07 14:48:26,821 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg164,60020,1302129205783
2011-04-07 14:48:26,822 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg163,60020,1302129205458
2011-04-07 14:48:26,822 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 1 region(s) to sv2borg165,60020,1302129205494
2011-04-07 14:48:26,822 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x2ec9c7920100ba Async create of unassigned node for
710a5516e0de5da60ed7ce9466ac3b6f with OFFLINE state
2011-04-07 14:48:26,823 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg164,60020,1302129205783
2011-04-07 14:48:26,823 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg166,60020,1302212905151
2011-04-07 14:48:26,823 DEBUG org.apache.hadoop.hbase.master.ServerManager: New
connection to sv2borg166,60020,1302212905151
2011-04-07 14:48:26,826 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg167,60020,1302129205469
2011-04-07 14:48:26,826 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rs=jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.
state=OFFLINE, ts=1302212906822, server=sv2borg165,60020,1302129205494
2011-04-07 14:48:26,826 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
assignment of regions to serverName=sv2borg166,60020,1302212905151,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0); bulk assign FAILED
java.io.IOException: Call to /10.20.20.166:60020 failed on local exception:
java.io.EOFException
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:806)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.openRegions(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:566)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:783)
at
org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1329)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:539)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:477)
2011-04-07 14:48:26,827 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg167,60020,1302129205469
2011-04-07 14:48:26,826 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback:
rs=jeichorn_test,,1302212906755.710a5516e0de5da60ed7ce9466ac3b6f.
state=OFFLINE, ts=1302212906822
2011-04-07 14:48:26,826 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning 0 region(s) to sv2borg168,60020,1302129205483
2011-04-07 14:48:26,829 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-04-07 14:48:26,829 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg168,60020,1302129205483
2011-04-07 14:48:26,829 INFO org.apache.hadoop.hbase.master.AssignmentManager:
sv2borg165,60020,1302129205494 unassigned znodes=1 of total=1
2011-04-07 14:48:26,829 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done for sv2borg165,60020,1302129205494
2011-04-07 14:48:26,995 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping
service threads
{code}
> createTable blocks until all regions are out of transition
> ----------------------------------------------------------
>
> Key: HBASE-3744
> URL: https://issues.apache.org/jira/browse/HBASE-3744
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.1
> Reporter: Todd Lipcon
> Assignee: Ted Yu
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 3744.txt
>
>
> In HBASE-3305, the behavior of createTable was changed and introduced this
> bug: createTable now blocks until all regions have been assigned, since it
> uses BulkStartupAssigner. BulkStartupAssigner.waitUntilDone calls
> assignmentManager.waitUntilNoRegionsInTransition, which waits across all
> regions, not just the regions of the table that has just been created.
> We saw an issue where one table had a region which was unable to be opened,
> so it was stuck in RegionsInTransition permanently (every open was failing).
> Since this was the case, waitUntilDone would always block indefinitely even
> though the newly created table had been assigned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
