Clément: Since you have reproduced the assertion error, can you log a JIRA ?
Thanks On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <[email protected]> wrote: > I modified the test so that the unit test runs against branch-1.4 : > > + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); > + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker"); > > However the test runs to completion without assertion - meaning the > assertion wouldn't be triggered in homogeneous cluster. > > Here were the last few lines in the test output: > > 2017-12-31 07:33:36,865 INFO [Time-limited test] > client.HBaseAdmin$CreateTableFuture(793): > Created AssertionTest > table created > 2017-12-31 07:33:36,871 DEBUG [Time-limited test] > client.ClientScanner(248): Advancing internal scanner to startKey at > '\x01\x02\x03\x04\x05\x06\x07', inclusive > 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] > ipc.RpcConnection(126): Use SIMPLE authentication for service > ClientService, sasl=false > 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3] > ipc.BlockingRpcConnection(431): Connecting to > tyus-mbp.attlocal.net/192.168.1.64:61585 > 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585] > ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from > 192.168.1.64:61633; # active connections: 3 > 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= > tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth > successful for tyu (auth:SIMPLE) > 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress= > tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839): > Connection from 192.168.1.64 port: 61633 with version info: version: > "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase" > revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date: > "Sun Dec 31 07:32:58 PST 2017" src_checksum: " > 39bdbbe40374e2094f53ce77af539671" version_major: 1 version_minor: 4 > 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after: > client.TestHCM#test Thread=325 (was 305) > > I assume you pointed the test at a 1.2.3 cluster which triggered the > assertion. > > It seems the patch which introduced the assertion didn't take into account > client running against older release cluster deployment. > > FYI > > On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <[email protected]> > wrote: > >> I have consistent failure with this code (a table with pre-splits): >> >> @Test >>> public void test() throws IOException{ >>> Configuration hbaseConfig = HBaseConfiguration.create(); >>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker"); >>> Connection connection = ConnectionFactory.createConnection(hbaseConfig); >>> >> >> >> TableName tableName = TableName.valueOf("AssertionTest"); >>> Admin admin = connection.getAdmin(); >>> >>> if(!admin.tableExists(tableName)){ >>> HTableDescriptor htable = new HTableDescriptor(tableName); >>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'})); >>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}}; >>> admin.createTable(htable, splitPoints); >>> System.out.println("table created"); >>> } >>> >>> Table table = connection.getTable(tableName); >>> ResultScanner scanner = table.getScanner(new Scan()); >>> scanner.iterator().hasNext(); // Exception thrown here >>> } >> >> >> 2017-12-30 5:04 GMT-08:00 Ted Yu <[email protected]>: >> >>> bq. a table has been created with pre-splits >>> >>> Unless you record the initial number of regions somewhere, there is no >>> built-in mechanism to provide the answer. >>> >>> Even if you cannot reproduce the assertion consistently, as long as the >>> chance of reproduction is high, the unit test would still be useful. >>> >>> Cheers >>> >>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume < >>> [email protected]> wrote: >>> >>>> I dropped all the tables in my test cluster. Now I'm inconstantly able >>>> to reproduce the issue. I'm digging more. >>>> >>>> I have a related question: is it possible to check if a table has been >>>> created with pre-splits? >>>> >>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but >>>> I think this should be ok according to this post >>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017 >>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E >>>> >>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <[email protected]>: >>>> >>>>> I got the following if I use your code directly: >>>>> >>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: >>>>> 0.096 sec <<< ERROR! >>>>> java.io.UncheckedIOException: >>>>> org.apache.hadoop.hbase.TableNotFoundException: >>>>> AssertionTest >>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243) >>>>> >>>>> If I create the table at the beginning of the test, the test passes >>>>> without triggering assertion. >>>>> >>>>> I tried both master branch and branch-1.4 >>>>> >>>>> As I mentioned previously, the assertion in your case was probably >>>>> related to the formation of row keys in hbase:meta table. >>>>> See if you can reproduce the assertion by presplitting the table, etc. >>>>> >>>>> Here is the diff I used based on branch-1.4 >>>>> >>>>> diff --git a/hbase-server/src/test/java/o >>>>> rg/apache/hadoop/hbase/client/TestHCM.java >>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>> TestHCM.java >>>>> index a5e3a65..2944c8e 100644 >>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>> TestHCM.java >>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/ >>>>> TestHCM.java >>>>> @@ -246,6 +246,18 @@ public class TestHCM { >>>>> TEST_UTIL.shutdownMiniCluster(); >>>>> } >>>>> >>>>> + @Test >>>>> + public void testAssert() throws IOException{ >>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration(); >>>>> + final TableName tableName = TableName.valueOf("assert"); >>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close(); >>>>> + String zkQuorum = "hbase123.docker"; >>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>> + Connection connection = ConnectionFactory.createConnec >>>>> tion(hbaseConfig); >>>>> + Table table = connection.getTable(tableName); >>>>> + ResultScanner scanner = table.getScanner(new Scan()); >>>>> + scanner.iterator().hasNext(); // Exception thrown here >>>>> + } >>>>> >>>>> private static int getHConnectionManagerCacheSize(){ >>>>> return HConnectionTestingUtility.getConnectionCount(); >>>>> >>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume < >>>>> [email protected]> wrote: >>>>> >>>>>> Here is a unit test, note that the AssertionTest table can exit or not >>>>>> exit, assertion will fail un both case. >>>>>> >>>>>> > @Test >>>>>> > public void test() throws IOException{ >>>>>> > Configuration hbaseConfig = HBaseConfiguration.create(); >>>>>> > String zkQuorum = "hbase123.docker"; >>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum); >>>>>> > Connection connection = ConnectionFactory.createConnec >>>>>> tion(hbaseConfig); >>>>>> > Table table = connection.getTable(TableName. >>>>>> valueOf("AssertionTest")); >>>>>> > ResultScanner scanner = table.getScanner(new Scan()); >>>>>> > scanner.iterator().hasNext(); // Exception thrown here >>>>>> > } >>>>>> > >>>>>> >>>>>> PS: I'm not receiving email when a response is posted here. I'm not >>>>>> subscribed at the hbase-user mailing list but I would have hope that >>>>>> answer >>>>>> get sent to me as direct email. >>>>>> >>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <[email protected]>: >>>>>> >>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system >>>>>> assertion >>>>>> > failure (in a failsafe test that run with system assertion enable) >>>>>> > >>>>>> > java.lang.AssertionError >>>>>> > at org.apache.hadoop.hbase.client.ClientScanner. >>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja >>>>>> r:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan >>>>>> ner.java:312) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag >>>>>> er.java:1324) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>> reImpl. >>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu >>>>>> reImpl. >>>>>> > groupAndSendMultiAction(AsyncProcess.java:980) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$ >>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>> rocess.java:649) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP >>>>>> rocess.java:612) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$ >>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan >>>>>> ager.java:2392) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j >>>>>> ava:950) >>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0] >>>>>> > >>>>>> > I have one Put action that look like that : >>>>>> {"totalColumns":1,"row":" >>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\ >>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":" >>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}} >>>>>> > The cluster is a local standalone instance. >>>>>> > >>>>>> > I haven't dig into it yet, but it looks like the data is correctly >>>>>> written >>>>>> > in hbase if I disable assertion. >>>>>> > Any idea of what's causing the issue? >>>>>> > >>>>>> > Thank you. >>>>>> > >>>>>> >>>>> >>>>> >>>> >>> >> >
