[ https://issues.apache.org/jira/browse/PHOENIX-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107973#comment-16107973 ]
James Taylor commented on PHOENIX-4052: --------------------------------------- The following test, slightly different than the description above, reproduces an issue with an extra row with an invalid row key ending up in the index table: {code} @Test public void testOutOfOrderDelete() throws Exception { String tableName = generateUniqueName(); String indexName = generateUniqueName(); Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES); long ts = 1000; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); Connection conn = DriverManager.getConnection(getUrl(), props); conn.createStatement().execute("CREATE TABLE " + tableName + "(k CHAR(1) PRIMARY KEY, v VARCHAR) COLUMN_ENCODED_BYTES = 0"); conn.close(); ts = 1010; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); conn = DriverManager.getConnection(getUrl(), props); conn.createStatement().execute("CREATE INDEX " + indexName + " ON " + tableName + "(v)"); conn.close(); ts = 1020; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); conn = DriverManager.getConnection(getUrl(), props); conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','a')"); conn.commit(); conn.close(); ts = 1040; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); conn = DriverManager.getConnection(getUrl(), props); conn.createStatement().execute("DELETE FROM " + tableName + " WHERE k='a'"); conn.commit(); conn.close(); ts = 1030; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); conn = DriverManager.getConnection(getUrl(), props); conn.createStatement().execute("UPSERT INTO " + tableName + " VALUES('a','bbb')"); conn.commit(); conn.close(); TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(tableName))); TestUtil.dumpTable(conn.unwrap(PhoenixConnection.class).getQueryServices().getTable(Bytes.toBytes(indexName))); ts = 1050; props.setProperty(PhoenixRuntime.CURRENT_SCN_ATTRIB, Long.toString(ts)); conn = DriverManager.getConnection(getUrl(), props); long count1 = getRowCount(conn, tableName); long count2 = getRowCount(conn, indexName); assertTrue("Expected table row count ( " + count1 + ") to match index row count (" + count2 + ")", count1 == count2); conn.close(); /** * ************ dumping T000001;hconnection-0x8f57e4c ************** a/0:/1040/DeleteFamily/vlen=0/seqid=0 a/0:V/1030/Put/vlen=3/seqid=0 a/0:V/1020/Put/vlen=1/seqid=0 a/0:_0/1030/Put/vlen=1/seqid=0 a/0:_0/1020/Put/vlen=1/seqid=0 ----------------------------------------------- ************ dumping T000002;hconnection-0x8f57e4c ************** \x00a/0:_0/1040/Put/vlen=2/seqid=0 a\x00a/0:/1040/DeleteFamily/vlen=0/seqid=0 a\x00a/0:/1030/DeleteFamily/vlen=0/seqid=0 a\x00a/0:_0/1020/Put/vlen=2/seqid=0 ----------------------------------------------- */ } private static long getRowCount(Connection conn, String tableName) throws SQLException { ResultSet rs = conn.createStatement().executeQuery("SELECT /*+ NO_INDEX */ count(*) FROM " + tableName); assertTrue(rs.next()); return rs.getLong(1); } {code} Assertion that occurs is: {code} java.lang.AssertionError: Expected table row count ( 0) to match index row count (1) at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.phoenix.end2end.index.OutOfOrderMutationsIT.testOutOfOrderDelete(OutOfOrderMutationsIT.java:135) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) {code} > Create the correct index row mutations for out-of-order data mutations > ---------------------------------------------------------------------- > > Key: PHOENIX-4052 > URL: https://issues.apache.org/jira/browse/PHOENIX-4052 > Project: Phoenix > Issue Type: Bug > Reporter: James Taylor > > Out-of-order processing of data rows during index maintenance causes mutable > indexes to become out of sync with regard to the data table. Here's a simple > example to illustrate the issue: > # Assume table T(K,V) and index X(V,K). > # Upsert T(A, 1) at t10. Index updates: Put X(1,A) at t10. > # Upsert T(A, 3) at t30. Index updates: Delete X(1,A) at t29, Put X(3,A) at > t30. > # Upsert T(A,2) at t20. Index updates: Delete X(1,A) at t19, Put X(2,A) at > t20, Delete X(2,A) at t29 > Ideally, we'd want to remove the Delete X(1,A) at t29 since this isn't > correct in terms of timeline consistency, but we can't do that with HBase > without support for deleting/undoing Delete markers. > The above is not what is occurring. Instead, when T(A,2) comes in, the Put > X(2,A) will occur at t20, but the Delete won't occur. This causes more index > rows than data rows, essentially making it invalid. > See PHOENIX-4051 as a workaround for fixing this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)