Thanks, Clément On Wed, Jan 3, 2018 at 2:54 PM, Clément Guillaume <[email protected]> wrote:
> Done https://issues.apache.org/jira/browse/HBASE-19700 > > 2017-12-31 8:16 GMT-08:00 Ted Yu <[email protected]>: > >> Clément: >> Since you have reproduced the assertion error, can you log a JIRA ? >> >> Thanks >> >> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <[email protected]> wrote: >> >>> I modified the test so that the unit test runs against branch-1.4 : >>> >>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); >>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, >>> "hbase123.docker"); >>> >>> However the test runs to completion without assertion - meaning the >>> assertion wouldn't be triggered in homogeneous cluster. >>> >>> Here were the last few lines in the test output: >>> >>> 2017-12-31 07:33:36,865 INFO [Time-limited test] >>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest >>> table created >>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test] >>> client.ClientScanner(248): Advancing internal scanner to startKey at >>> '\x01\x02\x03\x04\x05\x06\x07', inclusive >>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] >>> ipc.RpcConnection(126): Use SIMPLE authentication for service >>> ClientService, sasl=false >>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] >>> ipc.BlockingRpcConnection(431): Connecting to >>> tyus-mbp.attlocal.net/192.168.1.64:61585 >>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585] >>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from >>> 192.168.1.64:61633; # active connections: 3 >>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= >>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth >>> successful for tyu (auth:SIMPLE) >>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= >>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839): >>> Connection from 192.168.1.64 port: 61633 with version info: version: >>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase" >>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date: >>> "Sun Dec 31 07:32:58 PST 2017" src_checksum: >>> "39bdbbe40374e2094f53ce77af539671" >>> version_major: 1 version_minor: 4 >>> 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after: >>> client.TestHCM#test Thread=325 (was 305) >>> >>> I assume you pointed the test at a 1.2.3 cluster which triggered the >>> assertion. >>> >>> It seems the patch which introduced the assertion didn't take into >>> account client running against older release cluster deployment. >>> >>> FYI >>> >>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume < >>> [email protected]> wrote: >>> >>>> I have consistent failure with this code (a table with pre-splits): >>>> >>>> @Test >>>>> public void test() throws IOException{ >>>>> Configuration hbaseConfig = HBaseConfiguration.create(); >>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker"); >>>>> Connection connection = ConnectionFactory.createConnec >>>>> tion(hbaseConfig); >>>>> >>>> >>>> >>>> TableName tableName = TableName.valueOf("AssertionTest"); >>>>> Admin admin = connection.getAdmin(); >>>>> >>>>> if(!admin.tableExists(tableName)){ >>>>> HTableDescriptor htable = new HTableDescriptor(tableName); >>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'})); >>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}}; >>>>> admin.createTable(htable, splitPoints); >>>>> System.out.println("table created"); >>>>> } >>>>> >>>>> Table table = connection.getTable(tableName); >>>>> ResultScanner scanner = table.getScanner(new Scan()); >>>>> scanner.iterator().hasNext(); // Exception thrown here >>>>> } >>>> >>>> >>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <[email protected]>: >>>> >>>>> bq. a table has been created with pre-splits >>>>> >>>>> Unless you record the initial number of regions somewhere, there is no >>>>> built-in mechanism to provide the answer. >>>>> >>>>> Even if you cannot reproduce the assertion consistently, as long as >>>>> the chance of reproduction is high, the unit test would still be useful. >>>>> >>>>> Cheers >>>>> >>>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume < >>>>> [email protected]> wrote: >>>>> >>>>>> I dropped all the tables in my test cluster. Now I'm inconstantly >>>>>> able to reproduce the issue. I'm digging more. >>>>>> >>>>>> I have a related question: is it possible to check if a table has >>>>>> been created with pre-splits? >>>>>> >>>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, >>>>>> but I think this should be ok according to this post >>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017 >>>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E >>>>>> >>>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <[email protected]>: >>>>>> >>>>>>> I got the following if I use your code directly: >>>>>>> >>>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: >>>>>>> 0.096 sec <<< ERROR! >>>>>>> java.io.UncheckedIOException: >>>>>>> org.apache.hadoop.hbase.TableNotFoundException: >>>>>>> AssertionTest >>>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja >>>>>>> va:243) >>>>>>> >>>>>>> If I create the table at the beginning of the test, the test passes >>>>>>> without triggering assertion. >>>>>>> >>>>>>> I tried both master branch and branch-1.4 >>>>>>> >>>>>>> As I mentioned previously, the assertion in your case was probably >>>>>>> related to the formation of row keys in hbase:meta table. >>>>>>> See if you can reproduce the assertion by presplitting the table, >>>>>>> etc. >>>>>>> >>>>>>> Here is the diff I used based on branch-1.4 >>>>>>> >>>>>>> diff --git a/hbase-server/src/test/java/o >>>>>>> rg/apache/hadoop/hbase/client/TestHCM.java >>>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>>> TestHCM.java >>>>>>> index a5e3a65..2944c8e 100644 >>>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>>> TestHCM.java >>>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>>>> TestHCM.java >>>>>>> @@ -246,6 +246,18 @@ public class TestHCM { >>>>>>> TEST_UTIL.shutdownMiniCluster(); >>>>>>> } >>>>>>> >>>>>>> + @Test >>>>>>> + public void testAssert() throws IOException{ >>>>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); >>>>>>> + final TableName tableName = TableName.valueOf("assert"); >>>>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close(); >>>>>>> + String zkQuorum = "hbase123.docker"; >>>>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>>>> + Connection connection = ConnectionFactory.createConnec >>>>>>> tion(hbaseConfig); >>>>>>> + Table table = connection.getTable(tableName); >>>>>>> + ResultScanner scanner = table.getScanner(new Scan()); >>>>>>> + scanner.iterator().hasNext(); // Exception thrown here >>>>>>> + } >>>>>>> >>>>>>> private static int getHConnectionManagerCacheSize(){ >>>>>>> return HConnectionTestingUtility.getConnectionCount(); >>>>>>> >>>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Here is a unit test, note that the AssertionTest table can exit or >>>>>>>> not >>>>>>>> exit, assertion will fail un both case. >>>>>>>> >>>>>>>> > @Test >>>>>>>> > public void test() throws IOException{ >>>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create(); >>>>>>>> > String zkQuorum = "hbase123.docker"; >>>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>>>>> > Connection connection = ConnectionFactory.createConnec >>>>>>>> tion(hbaseConfig); >>>>>>>> > Table table = connection.getTable(TableName. >>>>>>>> valueOf("AssertionTest")); >>>>>>>> > ResultScanner scanner = table.getScanner(new Scan()); >>>>>>>> > scanner.iterator().hasNext(); // Exception thrown here >>>>>>>> > } >>>>>>>> > >>>>>>>> >>>>>>>> PS: I'm not receiving email when a response is posted here. I'm not >>>>>>>> subscribed at the hbase-user mailing list but I would have hope >>>>>>>> that answer >>>>>>>> get sent to me as direct email. >>>>>>>> >>>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <[email protected] >>>>>>>> >: >>>>>>>> >>>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system >>>>>>>> assertion >>>>>>>> > failure (in a failsafe test that run with system assertion enable) >>>>>>>> > >>>>>>>> > java.lang.AssertionError >>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner. >>>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja >>>>>>>> r:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan >>>>>>>> ner.java:312) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag >>>>>>>> er.java:1324) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav >>>>>>>> a:1221) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>>>> reImpl. >>>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>>>> reImpl. >>>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$ >>>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>>>> rocess.java:649) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>>>> rocess.java:612) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan >>>>>>>> ager.java:2392) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j >>>>>>>> ava:950) >>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>>>> > >>>>>>>> > I have one Put action that look like that : >>>>>>>> {"totalColumns":1,"row":" >>>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00 >>>>>>>> \\xFF\\ >>>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":" >>>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}} >>>>>>>> > The cluster is a local standalone instance. >>>>>>>> > >>>>>>>> > I haven't dig into it yet, but it looks like the data is >>>>>>>> correctly written >>>>>>>> > in hbase if I disable assertion. >>>>>>>> > Any idea of what's causing the issue? >>>>>>>> > >>>>>>>> > Thank you. >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
