[
https://issues.apache.org/jira/browse/HBASE-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774670#comment-13774670
]
Nick Dimiduk commented on HBASE-9639:
-------------------------------------
On the region server, you see the HFile group bulkloaded successfully into one
region and then an attempt to load the same file group into others. The source
files have been moved into place, to the file not found is thrown for
subsequent loads. Which Region loads the files appears to be non-deterministic
-- there's a race for whichever region can complete the load first.
{noformat}
2013-08-30 07:37:22,536 INFO [RpcServer.handler=5,port=60020]
regionserver.HStore: Validating hfile at
hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
for inclusion in store D region
IntegrationTestBulkLoad,,1377848151704.ba6689d409333b857c4c06f0b0485ca0.
2013-08-30 07:37:22,655 DEBUG [RpcServer.handler=5,port=60020]
access.SecureBulkLoadEndpoint: Moving
hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
to
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
2013-08-30 07:37:22,680 DEBUG [RpcServer.handler=5,port=60020]
regionserver.HRegionFileSystem: Committing store file
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
as
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_
2013-08-30 07:37:22,731 INFO [RpcServer.handler=5,port=60020]
regionserver.HStore: Loaded HFile
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
into store 'D' as
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_
- updating store file list.
2013-08-30 07:37:22,732 INFO [RpcServer.handler=5,port=60020]
regionserver.HStore: Successfully loaded store file
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
into store D (new location:
hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_)
2013-08-30 07:37:22,732 DEBUG [RpcServer.handler=5,port=60020]
access.SecureBulkLoadEndpoint: Bulk Load done for:
hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
2013-08-30 07:37:22,911 INFO [RpcServer.handler=8,port=60020]
regionserver.HStore: Validating hfile at
hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
for inclusion in store D region
IntegrationTestBulkLoad,x\x00\x00\x00\x00\x00\x00\x00,1377848151709.925bd31a52e5b70a61c2216b5c1d3683.
2013-08-30 07:37:22,914 INFO [RpcServer.handler=4,port=60020]
regionserver.HStore: Validating hfile at
hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
for inclusion in store D region
IntegrationTestBulkLoad,@\x00\x00\x00\x00\x00\x00\x00,1377848151707.57e30d9da6cf9af4763fe9968dfcf564.
org.apache.hadoop.io.MultipleIOException: 3 exceptions
[java.io.FileNotFoundException: File does not exist:
/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
...
{noformat}
> SecureBulkLoad dispatches file load requests to all Regions
> -----------------------------------------------------------
>
> Key: HBASE-9639
> URL: https://issues.apache.org/jira/browse/HBASE-9639
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
> Affects Versions: 0.95.2
> Environment: Hadoop2, Kerberos
> Reporter: Nick Dimiduk
>
> When running a bulk load on a secure environment and loading data into the
> first region of a table, the request to load the HFile set is dispatched to
> all Regions for the table. This is reproduced consistently by running
> IntegrationTestBulkLoad on a secure cluster. The load fails with an exception
> that looks like:
> {noformat}
> 2013-08-30 07:37:22,993 INFO [main] mapreduce.LoadIncrementalHFiles: Split
> occured while grouping HFiles, retry attempt 1 with 3 files remaining to
> group or split
> 2013-08-30 07:37:22,999 ERROR [main] mapreduce.LoadIncrementalHFiles:
> IOException during splitting
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/L/bf41ea13997b4e228d05e67ba7b1b686
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:403)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:284)
> at
> org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.runLinkedListMRJob(IntegrationTestBulkLoad.java:200)
> at
> org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.testBulkLoad(IntegrationTestBulkLoad.java:133)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
