Can you try 0.90.5 release which has offline Meta builder ? Cheers
On Dec 30, 2011, at 11:50 AM, Vladimir Rodionov <[email protected]> wrote: > This is stack trace of exception we got during routine flow execution. Flow > failed. > > > [2011-12-30 12:32:16] (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] > (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! > INFO [2011-12-30 04:32:16] (ExecUtil.java:262) - DEBUG [2011-12-30 12:32:16] > (HConnectionManager.java:1123) - Retry 19, sleep for 256000ms! > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > > -- skipped -- > > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - DEBUG [2011-12-30 12:36:32] > (MetaScanner.java:166) - Scanning .META. starting at > row=MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,00000000000000 > for max=10 rows > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - ERROR [2011-12-30 12:36:32] > (FinalizeOutputStage.java:142) - execute: Stage FinalizeOutputStage FAILED > with exception: Failed 19 actions: WrongRegionException: 19 times, servers > with issues: us01-ciqps1-grid03.carrieriq.com:60020, > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed > 19 actions: WrongRegionException: 19 times, servers with issues: > us01-ciqps1-grid03.carrieriq.com:60020, > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1239) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1253) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:836) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flushTable(StripedHBaseTable.java:426) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.mmp2.input.StripedHBaseTable.flush(StripedHBaseTable.java:452) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.mmp3.output.hbase.FactsToHBaseJob.commitHBaseTables(FactsToHBaseJob.java:327) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.stages.OutputStage.commitJobs(OutputStage.java:681) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.stages.FinalizeOutputStage.doExecute(FinalizeOutputStage.java:140) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.stages.OutputStage.execute(OutputStage.java:745) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:226) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.platform.flows.FlowExecutor.submitWorkflow(FlowExecutor.java:464) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.start(FlowExecutionTool.java:239) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.exec(FlowExecutionTool.java:169) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.oneRun(FlowExecutionTool.java:545) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.mainRunResult(FlowExecutionTool.java:580) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.mainRun(FlowExecutionTool.java:592) > INFO [2011-12-30 04:36:32] (ExecUtil.java:262) - at > com.carrieriq.m2m.flows.FlowExecutionTool.main(FlowExecutionTool.java:596) > > > 1. hbck found no issues > > but > > 2. check_meta.rb found 3 holes in .META > > 1/12/30 19:25:46 WARN check_meta: hole after REGION => {NAME => > 'HANG_BUG-PACKAGEINDEX,00000000000000000000000000000000,1324603628841.6a25d283cad8f4ef7e9e971a2ec8f931.', > STARTKEY => '00000000000000000000000000000000', ENDKEY => > '7\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C\xE27\x8C', > ENCODED => 6a25d283cad8f4ef7e9e971a2ec8f931, TABLE => {{NAME => > 'HANG_BUG-PACKAGEINDEX', FAMILIES => [{NAME => 'i', BLOOMFILTER => 'NONE', > REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '1', TTL => > '31536000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => > 'true'}, {NAME => 'u', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', > COMPRESSION => 'GZ', VERSIONS => '1', TTL => '31536000', BLOCKSIZE => > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} > 11/12/30 19:25:52 WARN check_meta: hole after REGION => {NAME => > 'M2M-INTEGRATION-TEST_FILTERED_OUTPUT-1324603526438,,1324603527798.7cc06fb7afc0036d4c1964735117d224.', > STARTKEY => '', ENDKEY => '00000000000000000000000000000000', ENCODED => > 7cc06fb7afc0036d4c1964735117d224, TABLE => {{NAME => > 'M2M-INTEGRATION-TEST_FILTERED_OUTPUT-1324603526438', FAMILIES => [{NAME => > 'd', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', > VERSIONS => '2', TTL => '2147472000', BLOCKSIZE => '65536', IN_MEMORY => > 'false', BLOCKCACHE => 'true'}, {NAME => 'i', BLOOMFILTER => 'ROW', > REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '2', TTL => > '2147472000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => > 'true'}, {NAME => 'v', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', > COMPRESSION => 'GZ', VERSIONS => '2', TTL => '2147472000', BLOCKSIZE => > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} > 11/12/30 19:25:52 WARN check_meta: hole after REGION => {NAME => > 'MetaTable,>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8,1324603528925.6cc13be6a134847ff7f2bda5bae0b7c5.', > STARTKEY => > '>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE9\x94>\xE8', > ENDKEY => 'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF', ENCODED => > 6cc13be6a134847ff7f2bda5bae0b7c5, TABLE => {{NAME => 'MetaTable', FAMILIES => > [{NAME => 'm', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => > 'GZ', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY > => 'false', BLOCKCACHE => 'true'}]}} > 11/12/30 19:25:54 DEBUG client.HTable$ClientScanner: Finished with scanning > at REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => > 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES => > [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS > => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', > IN_MEMORY => 'true', BLOCKCACHE => 'true'}]}} > 11/12/30 19:25:54 INFO check_meta: .META. has holes > > The most interesting part - is missing region in 'MetaTable' (this is was > original exception) > > 1. Is there are anything we could do to repair at least 'MetaTable'? > > 2. '.META.' inconsistency is a a most dangerous thing in a production > because it makes everything unusable (without low-level hackerish voodoo) . > Any plans to make meta operations transactional? > > > > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > > > Confidentiality Notice: The information contained in this message, including > any attachments hereto, may be confidential and is intended to be read only > by the individual or entity to whom this message is addressed. If the reader > of this message is not the intended recipient or an agent or designee of the > intended recipient, please note that any review, use, disclosure or > distribution of this message or its attachments, in any form, is strictly > prohibited. If you have received this message in error, please immediately > notify the sender and/or [email protected] and delete or destroy > any copy of this message and its attachments.
