Done https://issues.apache.org/jira/browse/HBASE-19700
2017-12-31 8:16 GMT-08:00 Ted Yu <[email protected]>: > Clément: > Since you have reproduced the assertion error, can you log a JIRA ? > > Thanks > > On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <[email protected]> wrote: > >> I modified the test so that the unit test runs against branch-1.4 : >> >> + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); >> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker"); >> >> However the test runs to completion without assertion - meaning the >> assertion wouldn't be triggered in homogeneous cluster. >> >> Here were the last few lines in the test output: >> >> 2017-12-31 07:33:36,865 INFO [Time-limited test] >> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest >> table created >> 2017-12-31 07:33:36,871 DEBUG [Time-limited test] >> client.ClientScanner(248): Advancing internal scanner to startKey at >> '\x01\x02\x03\x04\x05\x06\x07', inclusive >> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] >> ipc.RpcConnection(126): Use SIMPLE authentication for service >> ClientService, sasl=false >> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] >> ipc.BlockingRpcConnection(431): Connecting to >> tyus-mbp.attlocal.net/192.168.1.64:61585 >> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585] >> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from >> 192.168.1.64:61633; # active connections: 3 >> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= >> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth >> successful for tyu (auth:SIMPLE) >> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= >> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839): >> Connection from 192.168.1.64 port: 61633 with version info: version: >> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase" >> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date: >> "Sun Dec 31 07:32:58 PST 2017" src_checksum: >> "39bdbbe40374e2094f53ce77af539671" >> version_major: 1 version_minor: 4 >> 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after: >> client.TestHCM#test Thread=325 (was 305) >> >> I assume you pointed the test at a 1.2.3 cluster which triggered the >> assertion. >> >> It seems the patch which introduced the assertion didn't take into >> account client running against older release cluster deployment. >> >> FYI >> >> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <[email protected] >> > wrote: >> >>> I have consistent failure with this code (a table with pre-splits): >>> >>> @Test >>>> public void test() throws IOException{ >>>> Configuration hbaseConfig = HBaseConfiguration.create(); >>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker"); >>>> Connection connection = ConnectionFactory.createConnec >>>> tion(hbaseConfig); >>>> >>> >>> >>> TableName tableName = TableName.valueOf("AssertionTest"); >>>> Admin admin = connection.getAdmin(); >>>> >>>> if(!admin.tableExists(tableName)){ >>>> HTableDescriptor htable = new HTableDescriptor(tableName); >>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'})); >>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}}; >>>> admin.createTable(htable, splitPoints); >>>> System.out.println("table created"); >>>> } >>>> >>>> Table table = connection.getTable(tableName); >>>> ResultScanner scanner = table.getScanner(new Scan()); >>>> scanner.iterator().hasNext(); // Exception thrown here >>>> } >>> >>> >>> 2017-12-30 5:04 GMT-08:00 Ted Yu <[email protected]>: >>> >>>> bq. a table has been created with pre-splits >>>> >>>> Unless you record the initial number of regions somewhere, there is no >>>> built-in mechanism to provide the answer. >>>> >>>> Even if you cannot reproduce the assertion consistently, as long as the >>>> chance of reproduction is high, the unit test would still be useful. >>>> >>>> Cheers >>>> >>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume < >>>> [email protected]> wrote: >>>> >>>>> I dropped all the tables in my test cluster. Now I'm inconstantly able >>>>> to reproduce the issue. I'm digging more. >>>>> >>>>> I have a related question: is it possible to check if a table has been >>>>> created with pre-splits? >>>>> >>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, >>>>> but I think this should be ok according to this post >>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017 >>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E >>>>> >>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <[email protected]>: >>>>> >>>>>> I got the following if I use your code directly: >>>>>> >>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: >>>>>> 0.096 sec <<< ERROR! >>>>>> java.io.UncheckedIOException: >>>>>> org.apache.hadoop.hbase.TableNotFoundException: >>>>>> AssertionTest >>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja >>>>>> va:243) >>>>>> >>>>>> If I create the table at the beginning of the test, the test passes >>>>>> without triggering assertion. >>>>>> >>>>>> I tried both master branch and branch-1.4 >>>>>> >>>>>> As I mentioned previously, the assertion in your case was probably >>>>>> related to the formation of row keys in hbase:meta table. >>>>>> See if you can reproduce the assertion by presplitting the table, etc. >>>>>> >>>>>> Here is the diff I used based on branch-1.4 >>>>>> >>>>>> diff --git a/hbase-server/src/test/java/o >>>>>> rg/apache/hadoop/hbase/client/TestHCM.java >>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>> TestHCM.java >>>>>> index a5e3a65..2944c8e 100644 >>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>> TestHCM.java >>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>> TestHCM.java >>>>>> @@ -246,6 +246,18 @@ public class TestHCM { >>>>>> TEST_UTIL.shutdownMiniCluster(); >>>>>> } >>>>>> >>>>>> + @Test >>>>>> + public void testAssert() throws IOException{ >>>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); >>>>>> + final TableName tableName = TableName.valueOf("assert"); >>>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close(); >>>>>> + String zkQuorum = "hbase123.docker"; >>>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>>> + Connection connection = ConnectionFactory.createConnec >>>>>> tion(hbaseConfig); >>>>>> + Table table = connection.getTable(tableName); >>>>>> + ResultScanner scanner = table.getScanner(new Scan()); >>>>>> + scanner.iterator().hasNext(); // Exception thrown here >>>>>> + } >>>>>> >>>>>> private static int getHConnectionManagerCacheSize(){ >>>>>> return HConnectionTestingUtility.getConnectionCount(); >>>>>> >>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Here is a unit test, note that the AssertionTest table can exit or >>>>>>> not >>>>>>> exit, assertion will fail un both case. >>>>>>> >>>>>>> > @Test >>>>>>> > public void test() throws IOException{ >>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create(); >>>>>>> > String zkQuorum = "hbase123.docker"; >>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>>>> > Connection connection = ConnectionFactory.createConnec >>>>>>> tion(hbaseConfig); >>>>>>> > Table table = connection.getTable(TableName. >>>>>>> valueOf("AssertionTest")); >>>>>>> > ResultScanner scanner = table.getScanner(new Scan()); >>>>>>> > scanner.iterator().hasNext(); // Exception thrown here >>>>>>> > } >>>>>>> > >>>>>>> >>>>>>> PS: I'm not receiving email when a response is posted here. I'm not >>>>>>> subscribed at the hbase-user mailing list but I would have hope that >>>>>>> answer >>>>>>> get sent to me as direct email. >>>>>>> >>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <[email protected] >>>>>>> >: >>>>>>> >>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system >>>>>>> assertion >>>>>>> > failure (in a failsafe test that run with system assertion enable) >>>>>>> > >>>>>>> > java.lang.AssertionError >>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner. >>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja >>>>>>> r:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan >>>>>>> ner.java:312) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag >>>>>>> er.java:1324) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav >>>>>>> a:1221) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>>> reImpl. >>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>>> reImpl. >>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$ >>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>>> rocess.java:649) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>>> rocess.java:612) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan >>>>>>> ager.java:2392) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j >>>>>>> ava:950) >>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>> > >>>>>>> > I have one Put action that look like that : >>>>>>> {"totalColumns":1,"row":" >>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00 >>>>>>> \\xFF\\ >>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":" >>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}} >>>>>>> > The cluster is a local standalone instance. >>>>>>> > >>>>>>> > I haven't dig into it yet, but it looks like the data is correctly >>>>>>> written >>>>>>> > in hbase if I disable assertion. >>>>>>> > Any idea of what's causing the issue? >>>>>>> > >>>>>>> > Thank you. >>>>>>> > >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
