[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526114#comment-16526114 ] Jepson commented on PHOENIX-1718: - [~jamestaylor] Can you share these careful configuration ? > Unable to find cached index metadata during the stablity test with phoenix > -- > > Key: PHOENIX-1718 > URL: https://issues.apache.org/jira/browse/PHOENIX-1718 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.2.0 > Environment: linux os ( 128G ram,48T disk,24 cores) * 8 > Hadoop 2.5.1 > HBase 0.98.7 > Phoenix 4.2.1 >Reporter: wuchengzhi >Priority: Critical > Attachments: hbase-hadoop-regionserver-cluster-node134 .zip > > > I am making stablity test with phoenix 4.2.1 . But the regionserver became > very slow after 4 hours , and i found some error log in the regionserver log > file. > In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i > setup 2 regionserver in each pc (total 16 rs). > 1. create 8 tables, each table contains an index from TEST_USER0 to > TEST_USER7. > create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 > varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 > integer,attr8 integer,attr9 integer,attr10 integer ) > DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE > = '65536',SALT_BUCKETS=32; > create local index TEST_USER_INDEX0 on > TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); > > 2. deploy phoenix client each machine to upsert data to tables. ( client1 > upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) > One phoenix client start 6 threads, and each thread upsert 10,000 rows in > a batch. and each thread will upsert 500,000,000 in totally. > 8 clients ran in same time. > the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows > in hbase, and error occur frequently at about running 4 hours and 50 > minutes , and the rps became very slow , less than 10,000 (7, in normal) . > 2015-03-09 19:15:13,337 ERROR > [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: > Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: > ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. > key=-1715879467965695792 > region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. > Index update failed > java.util.concurrent.ExecutionException: > org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 > (INT10): Unable to find cached index metadata. key=-1715879467965695792 > region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. > Index update failed > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) > at > org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) > at > org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) > at > org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) > at > org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) > at > org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) > at > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2219) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4376) > at >
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482015#comment-14482015 ] James Taylor commented on PHOENIX-1718: --- FYI - [~mujtabachohan] the above is a good tip to improve perf. Maybe we can doc this somewhere? Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2.0 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical Attachments: hbase-hadoop-regionserver-cluster-node134 .zip I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2219) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4376) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368458#comment-14368458 ] wuchengzhi commented on PHOENIX-1718: - when i create tables without index, it's so fast that one day i can put 20,000,000,000. But when table with index, regionservers became very slow ,just 2,000,,000 totally. when with the index scenario i found the region compact too slow ,so there was many hfiles in the regions more than 10. i increase the datanode.handler(200) and regionserver.handler(100) , smallCompact threads(10). the compact it's much more better than before,and i can put the data from 2 billion up to 3.8 billion. but the tps seems not so good. just 20,000-40,000 , i expect 80,000 Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical Attachments: hbase-hadoop-regionserver-cluster-node134 .zip I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476)
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361622#comment-14361622 ] wuchengzhi commented on PHOENIX-1718: - the error appear even i reduce to 2000 rows per thread . now i add the maxServerCacheTimeToLiveMs and maxMetaDataCacheTimeToLiveMs to hbase-site.xml and what the value should be recommended? another question is ,about PHOENIX-1521,do you have some suggest to improve the performance property namephoenix.coprocessor.maxServerCacheTimeToLiveMs/name value180/value /property property namephoenix.coprocessor.maxMetaDataCacheTimeToLiveMs/name value180/value /property Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical Attachments: hbase-hadoop-regionserver-cluster-node134 .zip I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360473#comment-14360473 ] wuchengzhi commented on PHOENIX-1718: - i seems ok when i reduce to 2000 rows per thread. now it keeps on running for 12 hours and the error still didn't appear Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical Attachments: hbase-hadoop-regionserver-cluster-node134 .zip I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2219) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4376) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360621#comment-14360621 ] Rajeshbabu Chintaguntla commented on PHOENIX-1718: -- I think PHOENIX-1521 is the reason for slow ness which can happen at region server. To avoid the exception you can configure phoenix.coprocessor.maxServerCacheTimeToLiveMs with more value default it's 30 seconds. Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical Attachments: hbase-hadoop-regionserver-cluster-node134 .zip I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2219) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358434#comment-14358434 ] Rajeshbabu Chintaguntla commented on PHOENIX-1718: -- Hi [~bdifn] Is it possible to provide the region server logs when the writes become slow? Is something splits going on at that time? bq. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch You mean committing after upserting 10,000 rows? Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359811#comment-14359811
]
wuchengzhi commented on PHOENIX-1718:
-
Hi,[~rajeshbabu]
I upserting data like this.
int threads = 6;
final int rowBatch = 1;
final totalRows = 5;
ThreadPoolExecutor executor = new ThreadPoolExecutor(tn, tn, 30,
TimeUnit.SECONDS, new ArrayBlockingQueue(50));
for (int i = 0; i threads; i++) {
Connection conn = createConnection(server, port, db, user, passwd);
conn.setAutoCommit(false);
Statement stmt = conn.createStatement();
for (int j = 0; j totalRows; j++) {
String sql = createUpsertSql(db, table, ..., ...);
stmt.addBatch(sql);
if (j 0 j % (rowBatch) == 0) {
stmt.executeBatch();
conn.commit();
}
}
stmt.executeBatch();
conn.commit();
}
Unable to find cached index metadata during the stablity test with phoenix
--
Key: PHOENIX-1718
URL: https://issues.apache.org/jira/browse/PHOENIX-1718
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.2
Environment: linux os ( 128G ram,48T disk,24 cores) * 8
Hadoop 2.5.1
HBase 0.98.7
Phoenix 4.2.1
Reporter: wuchengzhi
Priority: Critical
Attachments: hbase-hadoop-regionserver-cluster-node134 .zip
I am making stablity test with phoenix 4.2.1 . But the regionserver became
very slow after 4 hours , and i found some error log in the regionserver log
file.
In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i
setup 2 regionserver in each pc (total 16 rs).
1. create 8 tables, each table contains an index from TEST_USER0 to
TEST_USER7.
create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2
varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7
integer,attr8 integer,attr9 integer,attr10 integer )
DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE
= '65536',SALT_BUCKETS=32;
create local index TEST_USER_INDEX0 on
TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10);
2. deploy phoenix client each machine to upsert data to tables. ( client1
upsert into TEST_USER0 , client 2 upsert into TEST_USER1.)
One phoenix client start 6 threads, and each thread upsert 10,000 rows in
a batch. and each thread will upsert 500,000,000 in totally.
8 clients ran in same time.
the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows
in hbase, and error occur frequently at about running 4 hours and 50
minutes , and the rps became very slow , less than 10,000 (7, in normal) .
2015-03-09 19:15:13,337 ERROR
[B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner:
Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException:
ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata.
key=-1715879467965695792
region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6.
Index update failed
java.util.concurrent.ExecutionException:
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008
(INT10): Unable to find cached index metadata. key=-1715879467965695792
region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6.
Index update failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
at
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
at
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
at
org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140)
at
org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274)
at
org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597)
at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356350#comment-14356350 ] James Taylor commented on PHOENIX-1718: --- [~rajeshbabu] - any ideas/advice? Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:877) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2476) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2263) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2215) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2219) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4376) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3580) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3469) at
[jira] [Commented] (PHOENIX-1718) Unable to find cached index metadata during the stablity test with phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354215#comment-14354215 ] wuchengzhi commented on PHOENIX-1718: - also, i add the configuration in hbase-site.xml !-- only for master with phoenix -- property namehbase.master.loadbalancer.class/name valueorg.apache.phoenix.hbase.index.balancer.IndexLoadBalancer/value /property property namehbase.coprocessor.master.classes/name valueorg.apache.phoenix.hbase.index.master.IndexMasterObserver/value /property !-- for regionserver with phoenix -- property namehbase.regionserver.wal.codec/name valueorg.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec/value /property property namehbase.region.server.rpc.scheduler.factory.class/name valueorg.apache.phoenix.hbase.index.ipc.PhoenixIndexRpcSchedulerFactory/value descriptionFactory to create the Phoenix RPC Scheduler that knows to put index updates into index queues/description /property Unable to find cached index metadata during the stablity test with phoenix -- Key: PHOENIX-1718 URL: https://issues.apache.org/jira/browse/PHOENIX-1718 Project: Phoenix Issue Type: Bug Affects Versions: 4.2 Environment: linux os ( 128G ram,48T disk,24 cores) * 8 Hadoop 2.5.1 HBase 0.98.7 Phoenix 4.2.1 Reporter: wuchengzhi Priority: Critical I am making stablity test with phoenix 4.2.1 . But the regionserver became very slow after 4 hours , and i found some error log in the regionserver log file. In this scenario,the cluster has 8 machines(128G ram, 24 cores , 48T disk). i setup 2 regionserver in each pc (total 16 rs). 1. create 8 tables, each table contains an index from TEST_USER0 to TEST_USER7. create table TEST_USER0 (id varchar primary key , attr1 varchar, attr2 varchar,attr3 varchar,attr4 varchar,attr5 varchar,attr6 integer,attr7 integer,attr8 integer,attr9 integer,attr10 integer ) DATA_BLOCK_ENCODING='FAST_DIFF',VERSIONS=1,BLOOMFILTER='ROW',COMPRESSION='LZ4',BLOCKSIZE = '65536',SALT_BUCKETS=32; create local index TEST_USER_INDEX0 on TEST5.TEST_USER0(attr1,attr2,attr3,attr4,attr5,attr6,attr7,attr8,attr9,attr10); 2. deploy phoenix client each machine to upsert data to tables. ( client1 upsert into TEST_USER0 , client 2 upsert into TEST_USER1.) One phoenix client start 6 threads, and each thread upsert 10,000 rows in a batch. and each thread will upsert 500,000,000 in totally. 8 clients ran in same time. the log as belowRunning 4 hours later, threre were about 1,000,000,000 rows in hbase, and error occur frequently at about running 4 hours and 50 minutes , and the rps became very slow , less than 10,000 (7, in normal) . 2015-03-09 19:15:13,337 ERROR [B.DefaultRpcServer.handler=2,queue=2,port=60022] parallel.BaseTaskRunner: Found a failed task because: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=-1715879467965695792 region=TEST5.TEST_USER6,\x08,1425881401238.aacbf69ea1156d403a4a54810cba15d6. Index update failed at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66) at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:140) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:274) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:203) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:881) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1522) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1597) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1554) at
