[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15121: -- Fix Version/s: (was: 3.0.0-beta-1) Resolution: Not A Problem Status: Resolved (was: Patch Available) There is no ConnectionImplementation on branch-3+ now. Resolve. Feel free to reopen if it is still a problem for branch-2.x. Thanks. > ConnectionImplementation#locateRegionInMeta() issue when master is restarted > > > Key: HBASE-15121 > URL: https://issues.apache.org/jira/browse/HBASE-15121 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Attachments: HBASE-15121-v0.patch, HBASE-15121-v0.patch > > > I notice this issue while i was running > IntegrationTestMTTR#testRestartMaster() test was failing on put operation. > Here is sequence of events from logs leading to failed put operation: > Master restart > {code} > INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh > hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | > cut -d ' ' -f2 | xargs kill -s SIGKILL"] > {code} > Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 > (this is additional logging inspecting metaKey which is used to search > hbase:meta ) > {code} > 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] > client.ConnectionImplementation: metaKey inspection: > table=IntegrationTestMTTRLoadTestTool row= > 70efdf2ec9b086079795c442636b55fb-17 metaKey= > IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > {code} > Client throwing TableNotFoundException (hbase:meta scan returned null) > {code} > 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] > client.ConnectionImplementation: regionInfo result is null: > HBaseWriterThread_5 throwing TableNotFoundException logging details > table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 > metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: > Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > And as result we have failed insert operation: > {code} > 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: > Failed to insert: 17 after 60046ms; region information: cached: > region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., > hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: > exception from null for 70efdf2ec9b086079795c442636b55fb-17 > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at >
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15121: -- Fix Version/s: (was: 2.0.0) 3.0.0 > ConnectionImplementation#locateRegionInMeta() issue when master is restarted > > > Key: HBASE-15121 > URL: https://issues.apache.org/jira/browse/HBASE-15121 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-15121-v0.patch, HBASE-15121-v0.patch > > > I notice this issue while i was running > IntegrationTestMTTR#testRestartMaster() test was failing on put operation. > Here is sequence of events from logs leading to failed put operation: > Master restart > {code} > INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh > hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | > cut -d ' ' -f2 | xargs kill -s SIGKILL"] > {code} > Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 > (this is additional logging inspecting metaKey which is used to search > hbase:meta ) > {code} > 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] > client.ConnectionImplementation: metaKey inspection: > table=IntegrationTestMTTRLoadTestTool row= > 70efdf2ec9b086079795c442636b55fb-17 metaKey= > IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > {code} > Client throwing TableNotFoundException (hbase:meta scan returned null) > {code} > 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] > client.ConnectionImplementation: regionInfo result is null: > HBaseWriterThread_5 throwing TableNotFoundException logging details > table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 > metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: > Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > And as result we have failed insert operation: > {code} > 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: > Failed to insert: 17 after 60046ms; region information: cached: > region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., > hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: > exception from null for 70efdf2ec9b086079795c442636b55fb-17 > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) >
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15121: Attachment: HBASE-15121-v0.patch Here is patch. > ConnectionImplementation#locateRegionInMeta() issue when master is restarted > > > Key: HBASE-15121 > URL: https://issues.apache.org/jira/browse/HBASE-15121 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15121-v0.patch > > > I notice this issue while i was running > IntegrationTestMTTR#testRestartMaster() test was failing on put operation. > Here is sequence of events from logs leading to failed put operation: > Master restart > {code} > INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh > hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | > cut -d ' ' -f2 | xargs kill -s SIGKILL"] > {code} > Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 > (this is additional logging inspecting metaKey which is used to search > hbase:meta ) > {code} > 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] > client.ConnectionImplementation: metaKey inspection: > table=IntegrationTestMTTRLoadTestTool row= > 70efdf2ec9b086079795c442636b55fb-17 metaKey= > IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > {code} > Client throwing TableNotFoundException (hbase:meta scan returned null) > {code} > 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] > client.ConnectionImplementation: regionInfo result is null: > HBaseWriterThread_5 throwing TableNotFoundException logging details > table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 > metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: > Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > And as result we have failed insert operation: > {code} > 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: > Failed to insert: 17 after 60046ms; region information: cached: > region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., > hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: > exception from null for 70efdf2ec9b086079795c442636b55fb-17 > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > leading to test failing: > {code} > Failed
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15121: Status: Patch Available (was: Open) > ConnectionImplementation#locateRegionInMeta() issue when master is restarted > > > Key: HBASE-15121 > URL: https://issues.apache.org/jira/browse/HBASE-15121 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15121-v0.patch > > > I notice this issue while i was running > IntegrationTestMTTR#testRestartMaster() test was failing on put operation. > Here is sequence of events from logs leading to failed put operation: > Master restart > {code} > INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh > hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | > cut -d ' ' -f2 | xargs kill -s SIGKILL"] > {code} > Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 > (this is additional logging inspecting metaKey which is used to search > hbase:meta ) > {code} > 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] > client.ConnectionImplementation: metaKey inspection: > table=IntegrationTestMTTRLoadTestTool row= > 70efdf2ec9b086079795c442636b55fb-17 metaKey= > IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > {code} > Client throwing TableNotFoundException (hbase:meta scan returned null) > {code} > 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] > client.ConnectionImplementation: regionInfo result is null: > HBaseWriterThread_5 throwing TableNotFoundException logging details > table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 > metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: > Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > And as result we have failed insert operation: > {code} > 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: > Failed to insert: 17 after 60046ms; region information: cached: > region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., > hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: > exception from null for 70efdf2ec9b086079795c442636b55fb-17 > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > leading to test failing: > {code} > Failed to write
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15121: Attachment: HBASE-15121-v0.patch First patch did not triggered build. Retry. > ConnectionImplementation#locateRegionInMeta() issue when master is restarted > > > Key: HBASE-15121 > URL: https://issues.apache.org/jira/browse/HBASE-15121 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15121-v0.patch, HBASE-15121-v0.patch > > > I notice this issue while i was running > IntegrationTestMTTR#testRestartMaster() test was failing on put operation. > Here is sequence of events from logs leading to failed put operation: > Master restart > {code} > INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh > hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | > cut -d ' ' -f2 | xargs kill -s SIGKILL"] > {code} > Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 > (this is additional logging inspecting metaKey which is used to search > hbase:meta ) > {code} > 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] > client.ConnectionImplementation: metaKey inspection: > table=IntegrationTestMTTRLoadTestTool row= > 70efdf2ec9b086079795c442636b55fb-17 metaKey= > IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > {code} > Client throwing TableNotFoundException (hbase:meta scan returned null) > {code} > 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] > client.ConnectionImplementation: regionInfo result is null: > HBaseWriterThread_5 throwing TableNotFoundException logging details > table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 > metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 > 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: > Failed to get region location > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) > {code} > And as result we have failed insert operation: > {code} > 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: > Failed to insert: 17 after 60046ms; region information: cached: > region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., > hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: > exception from null for 70efdf2ec9b086079795c442636b55fb-17 > org.apache.hadoop.hbase.TableNotFoundException: > IntegrationTestMTTRLoadTestTool > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) > at > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) > at > org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) >
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[ https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-15121: Description: I notice this issue while i was running IntegrationTestMTTR#testRestartMaster() test was failing on put operation. Here is sequence of events from logs leading to failed put operation: Master restart {code} INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGKILL"] {code} Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17 (this is additional logging inspecting metaKey which is used to search hbase:meta ) {code} 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9] client.ConnectionImplementation: metaKey inspection: table=IntegrationTestMTTRLoadTestTool row= 70efdf2ec9b086079795c442636b55fb-17 metaKey= IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 {code} Client throwing TableNotFoundException (hbase:meta scan returned null) {code} 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5] client.ConnectionImplementation: regionInfo result is null: HBaseWriterThread_5 throwing TableNotFoundException logging details table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17 metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) {code} And as result we have failed insert operation: {code} 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter: Failed to insert: 17 after 60046ms; region information: cached: region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7., hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors: exception from null for 70efdf2ec9b086079795c442636b55fb-17 org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569) at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146) at org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111) {code} leading to test failing: {code} Failed to write key: 17 2016-01-15 10:33:53,984 INFO [main] mttr.IntegrationTestMTTR: RestartMaster failed after 469878ms. java.util.concurrent.ExecutionException: java.lang.AssertionError: Load failed expected:<0> but was:<1> {code} Here is snippet from ConnectionImplementation#locateRegionInMeta() throwing exception: {code} try { Result regionInfoRow = null; ReversedClientScanner rcs = null; try { rcs = new ClientSmallReversedScanner(conf, s, TableName.META_TABLE_NAME, this, rpcCallerFactory, rpcControllerFactory, getMetaLookupPool(), 0); regionInfoRow = rcs.next(); } finally { if (rcs != null) { rcs.close(); }