[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746476#comment-15746476 ] Andrew Purtell commented on HBASE-17300: The client makes it appear its synchronous so this is not incorrect. I'm just letting you know on the master it is not. > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > Attachments: HBASE-17300.patch > > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746291#comment-15746291 ] Samarth Jain commented on HBASE-17300: -- Hmm. Sounds like the documentation needs to be updated? {code} /** * Creates a new table. * Synchronous operation. * * @param desc table descriptor for table * * @throws IllegalArgumentException if the table name is reserved * @throws MasterNotRunningException if master is not running * @throws TableExistsException if table already exists (If concurrent * threads, the table may have been created between test-for-existence * and attempt-at-creation). * @throws IOException if a remote or network exception occurs */ public void createTable(HTableDescriptor desc) {code} > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > Attachments: HBASE-17300.patch > > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746284#comment-15746284 ] Andrew Purtell commented on HBASE-17300: Table creation is not a synchronous operation. The client submits the admin action. The master asynchronously executes it. The client will check status via the meta table and spin and return when it sees that the table exists. When you have two threads concurrently submitting create requests, ignoring TableExistsException, and then issuing an op, this is fraught for races. I'd change the code to lift the admin action out. I'm going to resolve this issue as cannot reproduce. If you continue to have trouble with racy behavior with respect to table creation we can follow up on another issue targeted to that. > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > Attachments: HBASE-17300.patch > > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746258#comment-15746258 ] Samarth Jain commented on HBASE-17300: -- Thanks for taking a look, [~apurtell]. You are right the test passes consistently if I remove the table creation code from the Callable. So it does look like there is a race between table creation and and other operations on it. I can try and see if I can adjust the Phoenix upgrade code to not do the table creation. But it does sound like there is some bug lurking here since table creation is a sync operation? > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > Attachments: HBASE-17300.patch > > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744453#comment-15744453 ] Samarth Jain commented on HBASE-17300: -- I changed code in Phoenix land to pass null for existing value which is when I found this bug. The test fails with 0.98.17 too. I went back up to 0.98.15 and the test is failing there too. So looks like this issue has been around for a while. > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744438#comment-15744438 ] Samarth Jain commented on HBASE-17300: -- Thanks! Updated the patch to use TEST_UTIL.getHBaseAdmin(). Hopefully this will make it easy to add it in an IT test. > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.23, 1.2.4 >Reporter: Samarth Jain > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (HBaseAdmin admin = TEST_UTIL.getHBaseAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17300) Concurrently calling checkAndPut with expected value as null returns true unexpectedly
[ https://issues.apache.org/jira/browse/HBASE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743886#comment-15743886 ] Andrew Purtell commented on HBASE-17300: bq. I am using a bit of Phoenix API to get hold of HBaseAdmin. But it should be fairly straightforward to adopt it for HBase IT tests. You're more likely to get engagement from the HBase community if the repro case just works with the HBase API. FWIW > Concurrently calling checkAndPut with expected value as null returns true > unexpectedly > -- > > Key: HBASE-17300 > URL: https://issues.apache.org/jira/browse/HBASE-17300 > Project: HBase > Issue Type: Bug >Reporter: Samarth Jain > > Attached is the test case. I have added some comments so hopefully the test > makes sense. It actually is causing test failures on the Phoenix branches. > PS - I am using a bit of Phoenix API to get hold of HBaseAdmin. But it should > be fairly straightforward to adopt it for HBase IT tests. > The test fails consistently using HBase-0.98.23. It exhibits flappy behavior > with the 1.2 branch (failed twice in 5 tries). > {code} > @Test > public void testNullCheckAndPut() throws Exception { > try (Connection conn = DriverManager.getConnection(getUrl())) { > try (HBaseAdmin admin = > conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) { > Callable c1 = new CheckAndPutCallable(); > Callable c2 = new CheckAndPutCallable(); > ExecutorService e = Executors.newFixedThreadPool(5); > Future f1 = e.submit(c1); > Future f2 = e.submit(c2); > assertTrue(f1.get() || f2.get()); > assertFalse(f1.get() && f2.get()); > } > } > } > > > private static final class CheckAndPutCallable implements > Callable { > @Override > public Boolean call() throws Exception { > byte[] rowToLock = "ROW".getBytes(); > byte[] colFamily = "COLUMN_FAMILY".getBytes(); > byte[] column = "COLUMN".getBytes(); > byte[] newValue = "NEW_VALUE".getBytes(); > byte[] oldValue = "OLD_VALUE".getBytes(); > byte[] tableName = "table".getBytes(); > boolean acquired = false; > try (Connection conn = DriverManager.getConnection(getUrl())) { > try (HBaseAdmin admin = > conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin()) { > HTableDescriptor tableDesc = new > HTableDescriptor(TableName.valueOf(tableName)); > HColumnDescriptor columnDesc = new > HColumnDescriptor(colFamily); > columnDesc.setTimeToLive(600); > tableDesc.addFamily(columnDesc); > try { > admin.createTable(tableDesc); > } catch (TableExistsException e) { > // ignore > } > try (HTableInterface table = > admin.getConnection().getTable(tableName)) { > Put put = new Put(rowToLock); > put.add(colFamily, column, oldValue); // add a row > with column set to oldValue > table.put(put); > put = new Put(rowToLock); > put.add(colFamily, column, newValue); > // only one of the threads should be able to get > return value of true for the expected value of oldValue > acquired = table.checkAndPut(rowToLock, colFamily, > column, oldValue, put); > if (!acquired) { >// if a thread didn't get true before, then it > shouldn't get true this time either >// because the column DOES exist >acquired = table.checkAndPut(rowToLock, colFamily, > column, null, put); > } > } > } > } > return acquired; > } > } > {code} > cc [~apurtell], [~jamestaylor], [~lhofhansl]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)