This is stack trace of exception we got during routine flow execution. Flow
failed.
[2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16]
(HConnectionManager.java:1123) - Retry 19, sleep for 256000ms!
INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16]
(HConnectionManager.java:1123) - Retry 19, sleep for 256000ms!
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
-- skipped --
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32]
(MetaScanner.java:166) - Scanning .META. starting at
row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000
for max=10 rows
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - ERROR [2011-12-30 12:36:32]
(FinalizeOutputStage.java:142) - execute: Stage FinalizeOutputStage FAILED with
exception: Failed 19 actions: WrongRegionException: 19 times, servers with
issues: us01-ciqps1-grid03.carrieriq.com:60020,
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) -
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 19
actions: WrongRegionException: 19 times, servers with issues:
us01-ciqps1-grid03.carrieriq.com:60020,
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1239)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1253)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:836)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flushTable(StripedHBaseTable.java:426)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flush(StripedHBaseTable.java:452)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.mmp3.output.hbase.FactsToHBaseJob.commitHBaseTables(FactsToHBaseJob.java:327)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.stages.OutputStage.commitJobs(OutputStage.java:681)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.stages.FinalizeOutputStage.doExecute(FinalizeOutputStage.java:140)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.stages.OutputStage.execute(OutputStage.java:745)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:226)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:464)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.start(FlowExecutionTool.java:239)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.exec(FlowExecutionTool.java:169)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.oneRun(FlowExecutionTool.java:545)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.mainRunResult(FlowExecutionTool.java:580)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.mainRun(FlowExecutionTool.java:592)
INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at
com.carrieriq.m2m.flows.FlowExecutionTool.main(FlowExecutionTool.java:596)
1. hbck found no issues
but
2. check_meta.rb found 3 holes in .META
1/12/30 19:25:46 WARN check_meta: hole after REGION => {NAME =>
'HANG_BUG-PACKAGEINDEX,00000000000000000000000000000000,1324603628841.6a25d283cad8f4ef7e9e971a2ec8f931.',
STARTKEY => '00000000000000000000000000000000', ENDKEY =>
'7\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C',
ENCODED => 6a25d283cad8f4ef7e9e971a2ec8f931, TABLE => {{NAME =>
'HANG_BUG-PACKAGEINDEX', FAMILIES => [{NAME => 'i', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '1', TTL =>
'31536000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'u', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION =>
'GZ', VERSIONS => '1', TTL => '31536000', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
11/12/30 19:25:52 WARN check_meta: hole after REGION => {NAME =>
'M2M-INTEGRATION-TEST_FILTERED_OUTPUT-1324603526438,,1324603527798.7cc06fb7afc0036d4c1964735117d224.',
STARTKEY => '', ENDKEY => '00000000000000000000000000000000', ENCODED =>
7cc06fb7afc0036d4c1964735117d224, TABLE => {{NAME =>
'M2M-INTEGRATION-TEST_FILTERED_OUTPUT-1324603526438', FAMILIES => [{NAME =>
'd', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'GZ',
VERSIONS => '2', TTL => '2147472000', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}, {NAME => 'i', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '2', TTL =>
'2147472000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}, {NAME => 'v', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
COMPRESSION => 'GZ', VERSIONS => '2', TTL => '2147472000', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}
11/12/30 19:25:52 WARN check_meta: hole after REGION => {NAME =>
'MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,1324603528925.6cc13be6a134847ff7f2bda5bae0b7c5.',
STARTKEY =>
'>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8',
ENDKEY => 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF', ENCODED =>
6cc13be6a134847ff7f2bda5bae0b7c5, TABLE => {{NAME => 'MetaTable', FAMILIES =>
[{NAME => 'm', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION =>
'GZ', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}}
11/12/30 19:25:54 DEBUG client.HTable$ClientScanner: Finished with scanning at
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED =>
1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES => [{NAME
=> 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '10',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY =>
'true', BLOCKCACHE => 'true'}]}}
11/12/30 19:25:54 INFO check_meta: .META. has holes
The most interesting part - is missing region in 'MetaTable' (this is was
original exception)
1. Is there are anything we could do to repair at least 'MetaTable'?
2. '.META.' inconsistency is a a most dangerous thing in a production because
it makes everything unusable (without low-level hackerish voodoo) .
Any plans to make meta operations transactional?
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
Confidentiality Notice: The information contained in this message, including
any attachments hereto, may be confidential and is intended to be read only by
the individual or entity to whom this message is addressed. If the reader of
this message is not the intended recipient or an agent or designee of the
intended recipient, please note that any review, use, disclosure or
distribution of this message or its attachments, in any form, is strictly
prohibited. If you have received this message in error, please immediately
notify the sender and/or [email protected] and delete or destroy any
copy of this message and its attachments.
