Build failed in Jenkins: Phoenix | Master #2042

[Apache Jenkins Server]

See 


Changes:

[ankitsinghal59] PHOENIX-4768 Re-enable testCompactUpdatesStats and

[ankitsinghal59] PHOENIX-4769 Annotate SystemCatalogIT so that it will run with 
the test

--
[...truncated 103.84 KB...]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.735 s 
- in org.apache.phoenix.end2end.UpsertBigValuesIT
[INFO] Running org.apache.phoenix.end2end.UpsertSelectAutoCommitIT
[INFO] Running org.apache.phoenix.end2end.UpgradeIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.958 s 
- in org.apache.phoenix.end2end.UpsertSelectAutoCommitIT
[INFO] Running org.apache.phoenix.end2end.UpsertSelectIT
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 172.631 
s - in org.apache.phoenix.end2end.UnionAllIT
[INFO] Running org.apache.phoenix.end2end.UpsertValuesIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.871 
s - in org.apache.phoenix.end2end.UpsertValuesIT
[INFO] Running org.apache.phoenix.end2end.UseSchemaIT
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.497 s 
- in org.apache.phoenix.end2end.UseSchemaIT
[INFO] Running org.apache.phoenix.end2end.VariableLengthPKIT
[INFO] Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 204.247 
s - in org.apache.phoenix.end2end.UpsertSelectIT
[INFO] Running org.apache.phoenix.end2end.index.AsyncIndexDisabledIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.65 s - 
in org.apache.phoenix.end2end.index.AsyncIndexDisabledIT
[INFO] Running org.apache.phoenix.end2end.index.ChildViewsUseParentViewIndexIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.683 s 
- in org.apache.phoenix.end2end.index.ChildViewsUseParentViewIndexIT
[INFO] Running org.apache.phoenix.end2end.index.DropColumnIT
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 397.622 
s - in org.apache.phoenix.end2end.UpgradeIT
[INFO] Running org.apache.phoenix.end2end.index.DropMetadataIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.4 s - 
in org.apache.phoenix.end2end.index.DropMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
[INFO] Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 522.697 
s - in org.apache.phoenix.end2end.UngroupedIT
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 215.622 
s - in org.apache.phoenix.end2end.VariableLengthPKIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalIndexOptimizationIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.126 s 
- in org.apache.phoenix.end2end.index.GlobalIndexOptimizationIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalMutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalImmutableTxIndexIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 389.627 
s - in org.apache.phoenix.end2end.index.DropColumnIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 312.79 
s - in org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 327.508 
s - in org.apache.phoenix.end2end.index.GlobalMutableNonTxIndexIT
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.292 
s - in org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Running org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 140.282 
s - in org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 337.182 
s - in org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[INFO] Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 348.163 
s - in org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 229.975 
s - in org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableNonTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 430.089 
s - in org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 458.269 
s - in 

[2/2] phoenix git commit: PHOENIX-4768 Re-enable testCompactUpdatesStats and testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT

[ankit]

PHOENIX-4768 Re-enable testCompactUpdatesStats and 
testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/1ae275c1
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/1ae275c1
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/1ae275c1

Branch: refs/heads/4.x-HBase-0.98
Commit: 1ae275c15a65ce70c3ac7f00e6751b80dc270ca2
Parents: 9d29e90
Author: Ankit Singhal 
Authored: Fri Jun 1 14:45:45 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:45:45 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/1ae275c1/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 09d28f8..5436311 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -68,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -404,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -463,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -476,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[1/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-0.98 5ce06a062 -> 1ae275c15


PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/9d29e905
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/9d29e905
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/9d29e905

Branch: refs/heads/4.x-HBase-0.98
Commit: 9d29e9051766d65a3cc2412ea1be09f111602a0d
Parents: 5ce06a0
Author: Ankit Singhal 
Authored: Fri Jun 1 14:45:36 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:45:36 2018 -0700

--
 .../apache/phoenix/end2end/SystemCatalogIT.java   | 18 --
 1 file changed, 12 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/9d29e905/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index acc7873..c21c399 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -18,7 +18,6 @@
 package org.apache.phoenix.end2end;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
 
 import java.sql.Connection;
 import java.sql.DriverManager;
@@ -26,13 +25,17 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -55,10 +58,13 @@ public class SystemCatalogIT {
 }
 TableName systemCatalog = TableName.valueOf("SYSTEM.CATALOG");
 assertEquals(1, 
testUtil.getHBaseAdmin().getTableRegions(systemCatalog).size());
-
-// now attempt to split SYSTEM.CATALOG
-testUtil.getHBaseAdmin().split(systemCatalog.getName());
-
+try {
+// now attempt to split SYSTEM.CATALOG
+testUtil.getHBaseAdmin().split(systemCatalog.getName());
+} catch (DoNotRetryIOException e) {
+// table is not splittable
+assert (e.getMessage().contains("NOT splittable"));
+}
 // make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
 testUtil.getHBaseAdmin().disableTable(systemCatalog);
 testUtil.getHBaseAdmin().enableTable(systemCatalog);

[2/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/27a66607
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/27a66607
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/27a66607

Branch: refs/heads/4.x-HBase-1.1
Commit: 27a66607f3f67273367c7c5d2b1d20b721ec227b
Parents: 8657ce2
Author: Ankit Singhal 
Authored: Fri Jun 1 14:43:30 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:43:30 2018 -0700

--
 .../apache/phoenix/end2end/SystemCatalogIT.java | 23 +---
 1 file changed, 15 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/27a66607/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index 15af2af..2626ad6 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -19,7 +19,6 @@
 package org.apache.phoenix.end2end;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
 
 import java.sql.Connection;
 import java.sql.DriverManager;
@@ -27,14 +26,18 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -58,13 +61,17 @@ public class SystemCatalogIT {
 TableName systemCatalog = TableName.valueOf("SYSTEM.CATALOG");
 RegionLocator rl = 
testUtil.getConnection().getRegionLocator(systemCatalog);
 assertEquals(rl.getAllRegionLocations().size(), 1);
+try {
+// now attempt to split SYSTEM.CATALOG
+testUtil.getHBaseAdmin().split(systemCatalog);
 
-// now attempt to split SYSTEM.CATALOG
-testUtil.getHBaseAdmin().split(systemCatalog);
-
-// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
-testUtil.getHBaseAdmin().disableTable(systemCatalog);
-testUtil.getHBaseAdmin().enableTable(systemCatalog);
+// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
+testUtil.getHBaseAdmin().disableTable(systemCatalog);
+testUtil.getHBaseAdmin().enableTable(systemCatalog);
+} catch (DoNotRetryIOException e) {
+// table is not splittable
+assert (e.getMessage().contains("NOT splittable"));
+}
 
 // test again... Must still be exactly one region.
 rl = testUtil.getConnection().getRegionLocator(systemCatalog);

[1/2] phoenix git commit: PHOENIX-4768 Re-enable testCompactUpdatesStats and testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 f8d09637c -> 27a66607f


PHOENIX-4768 Re-enable testCompactUpdatesStats and 
testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/8657ce2c
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/8657ce2c
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/8657ce2c

Branch: refs/heads/4.x-HBase-1.1
Commit: 8657ce2cd2eca81d22ccaa78f4b2183a40f4935b
Parents: f8d0963
Author: Ankit Singhal 
Authored: Fri Jun 1 14:43:22 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:43:22 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/8657ce2c/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 09d28f8..5436311 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -68,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -404,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -463,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -476,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[2/2] phoenix git commit: PHOENIX-4768 Re-enable testCompactUpdatesStats and testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT

[ankit]

PHOENIX-4768 Re-enable testCompactUpdatesStats and 
testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/ce834bec
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/ce834bec
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/ce834bec

Branch: refs/heads/4.x-HBase-1.2
Commit: ce834bec462094a203e1c99b52ce82b8da89
Parents: 091ffbd
Author: Ankit Singhal 
Authored: Fri Jun 1 14:43:01 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:43:01 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/ce834bec/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 09d28f8..5436311 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -68,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -404,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -463,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -476,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[1/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 445f72750 -> ce834bec4


PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/091ffbd8
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/091ffbd8
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/091ffbd8

Branch: refs/heads/4.x-HBase-1.2
Commit: 091ffbd856275a95f826a7915db57e8918448e53
Parents: 445f727
Author: Ankit Singhal 
Authored: Fri Jun 1 14:42:50 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:42:50 2018 -0700

--
 .../apache/phoenix/end2end/SystemCatalogIT.java | 23 +---
 1 file changed, 15 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/091ffbd8/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index 7b6a543..6f49518 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -18,7 +18,6 @@
 package org.apache.phoenix.end2end;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
 
 import java.sql.Connection;
 import java.sql.DriverManager;
@@ -26,14 +25,18 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -57,13 +60,17 @@ public class SystemCatalogIT {
 TableName systemCatalog = TableName.valueOf("SYSTEM.CATALOG");
 RegionLocator rl = 
testUtil.getConnection().getRegionLocator(systemCatalog);
 assertEquals(rl.getAllRegionLocations().size(), 1);
+try {
+// now attempt to split SYSTEM.CATALOG
+testUtil.getHBaseAdmin().split(systemCatalog);
 
-// now attempt to split SYSTEM.CATALOG
-testUtil.getHBaseAdmin().split(systemCatalog);
-
-// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
-testUtil.getHBaseAdmin().disableTable(systemCatalog);
-testUtil.getHBaseAdmin().enableTable(systemCatalog);
+// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
+testUtil.getHBaseAdmin().disableTable(systemCatalog);
+testUtil.getHBaseAdmin().enableTable(systemCatalog);
+} catch (DoNotRetryIOException e) {
+// table is not splittable
+assert (e.getMessage().contains("NOT splittable"));
+}
 
 // test again... Must still be exactly one region.
 rl = testUtil.getConnection().getRegionLocator(systemCatalog);

[2/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/190ff840
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/190ff840
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/190ff840

Branch: refs/heads/4.x-HBase-1.3
Commit: 190ff8401a4b155ccf5a849aced0e1cbade92445
Parents: 2f67e9f
Author: Ankit Singhal 
Authored: Fri Jun 1 14:42:27 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:42:27 2018 -0700

--
 .../apache/phoenix/end2end/SystemCatalogIT.java | 23 +---
 1 file changed, 15 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/190ff840/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index 7b6a543..6f49518 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -18,7 +18,6 @@
 package org.apache.phoenix.end2end;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
 
 import java.sql.Connection;
 import java.sql.DriverManager;
@@ -26,14 +25,18 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -57,13 +60,17 @@ public class SystemCatalogIT {
 TableName systemCatalog = TableName.valueOf("SYSTEM.CATALOG");
 RegionLocator rl = 
testUtil.getConnection().getRegionLocator(systemCatalog);
 assertEquals(rl.getAllRegionLocations().size(), 1);
+try {
+// now attempt to split SYSTEM.CATALOG
+testUtil.getHBaseAdmin().split(systemCatalog);
 
-// now attempt to split SYSTEM.CATALOG
-testUtil.getHBaseAdmin().split(systemCatalog);
-
-// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
-testUtil.getHBaseAdmin().disableTable(systemCatalog);
-testUtil.getHBaseAdmin().enableTable(systemCatalog);
+// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
+testUtil.getHBaseAdmin().disableTable(systemCatalog);
+testUtil.getHBaseAdmin().enableTable(systemCatalog);
+} catch (DoNotRetryIOException e) {
+// table is not splittable
+assert (e.getMessage().contains("NOT splittable"));
+}
 
 // test again... Must still be exactly one region.
 rl = testUtil.getConnection().getRegionLocator(systemCatalog);

[1/2] phoenix git commit: PHOENIX-4768 Re-enable testCompactUpdatesStats and testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/master 14baca9de -> 877b724d6


PHOENIX-4768 Re-enable testCompactUpdatesStats and 
testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/98f7a69c
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/98f7a69c
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/98f7a69c

Branch: refs/heads/master
Commit: 98f7a69c8063ebee0c2072c27abd3a386a714ad3
Parents: 14baca9
Author: Ankit Singhal 
Authored: Fri Jun 1 14:25:05 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:25:05 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/98f7a69c/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 09d28f8..5436311 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -68,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -404,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -463,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -476,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (HTableInterface htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[2/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/877b724d
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/877b724d
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/877b724d

Branch: refs/heads/master
Commit: 877b724d6496957aacb70464e930a7e1cfed51e7
Parents: 98f7a69
Author: Ankit Singhal 
Authored: Fri Jun 1 14:27:22 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:27:22 2018 -0700

--
 .../apache/phoenix/end2end/SystemCatalogIT.java | 23 +---
 1 file changed, 15 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/877b724d/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index 7b6a543..6f49518 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -18,7 +18,6 @@
 package org.apache.phoenix.end2end;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
 
 import java.sql.Connection;
 import java.sql.DriverManager;
@@ -26,14 +25,18 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -57,13 +60,17 @@ public class SystemCatalogIT {
 TableName systemCatalog = TableName.valueOf("SYSTEM.CATALOG");
 RegionLocator rl = 
testUtil.getConnection().getRegionLocator(systemCatalog);
 assertEquals(rl.getAllRegionLocations().size(), 1);
+try {
+// now attempt to split SYSTEM.CATALOG
+testUtil.getHBaseAdmin().split(systemCatalog);
 
-// now attempt to split SYSTEM.CATALOG
-testUtil.getHBaseAdmin().split(systemCatalog);
-
-// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
-testUtil.getHBaseAdmin().disableTable(systemCatalog);
-testUtil.getHBaseAdmin().enableTable(systemCatalog);
+// make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
+testUtil.getHBaseAdmin().disableTable(systemCatalog);
+testUtil.getHBaseAdmin().enableTable(systemCatalog);
+} catch (DoNotRetryIOException e) {
+// table is not splittable
+assert (e.getMessage().contains("NOT splittable"));
+}
 
 // test again... Must still be exactly one region.
 rl = testUtil.getConnection().getRegionLocator(systemCatalog);

phoenix git commit: PHOENIX-4770 Re-enable pending tests which were ignored during 5.0.0 alpha

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 09c017e3c -> 3b1c0d317


PHOENIX-4770 Re-enable pending tests which were ignored during 5.0.0 alpha


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/3b1c0d31
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/3b1c0d31
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/3b1c0d31

Branch: refs/heads/5.x-HBase-2.0
Commit: 3b1c0d31743bc9d9a4b99d6a0d276c00ddbd837c
Parents: 09c017e
Author: Ankit Singhal 
Authored: Fri Jun 1 14:20:40 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:20:40 2018 -0700

--
 .../phoenix/end2end/IndexScrutinyToolIT.java|  8 +--
 .../phoenix/end2end/index/LocalIndexIT.java |  2 -
 .../schema/types/PDataTypeForArraysTest.java| 54 ++--
 3 files changed, 8 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/3b1c0d31/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
index d5c50a1..692a98c 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
@@ -40,7 +40,6 @@ import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 
-import com.google.common.collect.Sets;
 import org.apache.commons.io.IOUtils;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FSDataInputStream;
@@ -71,18 +70,18 @@ import org.junit.BeforeClass;
 import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
+import com.google.common.collect.Sets;
 
 /**
  * Tests for the {@link IndexScrutinyTool}
  */
 @Category(NeedsOwnMiniClusterTest.class)
 @RunWith(Parameterized.class)
-@Ignore
 public class IndexScrutinyToolIT extends BaseTest {
 
 private String dataTableDdl;
@@ -192,6 +191,7 @@ public class IndexScrutinyToolIT extends BaseTest {
  * Since CURRENT_SCN is set, the scrutiny shouldn't report any issue.
  */
 @Test
+@Ignore("PHOENIX-4378 Unable to set KEEP_DELETED_CELLS to true on RS 
scanner")
 public void testScrutinyWhileTakingWrites() throws Exception {
 int id = 0;
 while (id < 1000) {

http://git-wip-us.apache.org/repos/asf/phoenix/blob/3b1c0d31/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
index 15d938c..41616f2 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
@@ -71,7 +71,6 @@ import org.apache.phoenix.schema.TableNotFoundException;
 import org.apache.phoenix.util.QueryUtil;
 import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
-import org.junit.Ignore;
 import org.junit.Test;
 
 import com.google.common.collect.Lists;
@@ -587,7 +586,6 @@ public class LocalIndexIT extends BaseLocalIndexIT {
 }
 
 @Test
-@Ignore
 public void testLocalIndexAutomaticRepair() throws Exception {
 if (isNamespaceMapped) { return; }
 PhoenixConnection conn = 
DriverManager.getConnection(getUrl()).unwrap(PhoenixConnection.class);

http://git-wip-us.apache.org/repos/asf/phoenix/blob/3b1c0d31/phoenix-core/src/test/java/org/apache/phoenix/schema/types/PDataTypeForArraysTest.java
--
diff --git 
a/phoenix-core/src/test/java/org/apache/phoenix/schema/types/PDataTypeForArraysTest.java
 
b/phoenix-core/src/test/java/org/apache/phoenix/schema/types/PDataTypeForArraysTest.java
index 290c80f..792ec6b 100644
--- 
a/phoenix-core/src/test/java/org/apache/phoenix/schema/types/PDataTypeForArraysTest.java
+++ 
b/phoenix-core/src/test/java/org/apache/phoenix/schema/types/PDataTypeForArraysTest.java
@@ -17,6 +17,10 @@
  */
 package org.apache.phoenix.schema.types;
 
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static 

[1/2] phoenix git commit: This reverts commit 3a7c9033936779f5290f4fabdd4912c27f7af64f to just fix the JIRA No.

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 3a7c90339 -> 09c017e3c


This reverts commit 3a7c9033936779f5290f4fabdd4912c27f7af64f to just fix the 
JIRA No.


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/3df50d8c
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/3df50d8c
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/3df50d8c

Branch: refs/heads/5.x-HBase-2.0
Commit: 3df50d8c2a38f53e7c978bd0d36197167256e306
Parents: 3a7c903
Author: Ankit Singhal 
Authored: Fri Jun 1 14:15:14 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:15:14 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java| 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/3df50d8c/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index c2325ae..3af0d09 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -46,6 +46,7 @@ import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver;
@@ -68,6 +69,7 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
+import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -403,11 +405,13 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
+@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
+@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -460,7 +464,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -473,7 +477,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[2/2] phoenix git commit: PHOENIX-4768 Re-enable testCompactUpdatesStats and testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT

[ankit]

PHOENIX-4768 Re-enable testCompactUpdatesStats and 
testCompactUpdatesStatsWithMinStatsUpdateFreq of StatsCollectorIT


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/09c017e3
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/09c017e3
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/09c017e3

Branch: refs/heads/5.x-HBase-2.0
Commit: 09c017e3cc8a2d1ec4ce27c65a5c76b71de138a0
Parents: 3df50d8
Author: Ankit Singhal 
Authored: Fri Jun 1 14:17:58 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:17:58 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java| 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/09c017e3/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 3af0d09..c2325ae 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -46,7 +46,6 @@ import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.client.Table;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver;
@@ -69,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -405,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -464,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -477,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[2/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite

[ankit]

PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/3a7c9033
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/3a7c9033
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/3a7c9033

Branch: refs/heads/5.x-HBase-2.0
Commit: 3a7c9033936779f5290f4fabdd4912c27f7af64f
Parents: f8a1f1e
Author: Ankit Singhal 
Authored: Fri Jun 1 14:13:41 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:13:41 2018 -0700

--
 .../org/apache/phoenix/schema/stats/StatsCollectorIT.java| 8 ++--
 1 file changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/3a7c9033/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
index 3af0d09..c2325ae 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/schema/stats/StatsCollectorIT.java
@@ -46,7 +46,6 @@ import org.apache.hadoop.hbase.client.Result;
 import org.apache.hadoop.hbase.client.ResultScanner;
 import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.client.Table;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver;
@@ -69,7 +68,6 @@ import org.apache.phoenix.util.SchemaUtil;
 import org.apache.phoenix.util.TestUtil;
 import org.junit.Before;
 import org.junit.BeforeClass;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
@@ -405,13 +403,11 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStats() throws Exception {
 testCompactUpdatesStats(0, fullTableName);
 }
 
 @Test
-@Ignore //TODO remove this once  
https://issues.apache.org/jira/browse/TEPHRA-208 is fixed
 public void testCompactUpdatesStatsWithMinStatsUpdateFreq() throws 
Exception {
 
testCompactUpdatesStats(QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS, 
fullTableName);
 }
@@ -464,7 +460,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 Scan scan = new Scan();
 scan.setRaw(true);
 PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {
@@ -477,7 +473,7 @@ public abstract class StatsCollectorIT extends 
BaseUniqueNamesOwnClusterIT {
 scan = new Scan();
 scan.setRaw(true);
 phxConn = conn.unwrap(PhoenixConnection.class);
-try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(tableName))) {
+try (Table htable = 
phxConn.getQueryServices().getTable(Bytes.toBytes(physicalTableName))) {
 ResultScanner scanner = htable.getScanner(scan);
 Result result;
 while ((result = scanner.next())!=null) {

[1/2] phoenix git commit: PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite.

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 c489587e4 -> 3a7c90339


PHOENIX-4769 Annotate SystemCatalogIT so that it will run with the test suite.


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/f8a1f1ec
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/f8a1f1ec
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/f8a1f1ec

Branch: refs/heads/5.x-HBase-2.0
Commit: f8a1f1ece77a9684b72d04a2d25148cb044e56ce
Parents: c489587
Author: Ankit Singhal 
Authored: Fri Jun 1 14:12:22 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:12:22 2018 -0700

--
 .../org/apache/phoenix/end2end/SystemCatalogIT.java| 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/f8a1f1ec/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
index 9c66b9a..2c7b992 100644
--- a/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
+++ b/phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogIT.java
@@ -25,16 +25,18 @@ import java.sql.SQLException;
 import java.sql.Statement;
 import java.util.Properties;
 
+import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.phoenix.query.BaseTest;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.junit.After;
-import org.junit.Ignore;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 
-@Ignore
-public class SystemCatalogIT {
+@Category(NeedsOwnMiniClusterTest.class)
+public class SystemCatalogIT extends BaseTest {
 private HBaseTestingUtility testUtil = null;
 
 @After
@@ -59,12 +61,17 @@ public class SystemCatalogIT {
 RegionLocator rl = 
testUtil.getConnection().getRegionLocator(systemCatalog);
 assertEquals(rl.getAllRegionLocations().size(), 1);
 
+try{
 // now attempt to split SYSTEM.CATALOG
 testUtil.getAdmin().split(systemCatalog);
 
 // make sure the split finishes (there's no synchronous splitting 
before HBase 2.x)
 testUtil.getAdmin().disableTable(systemCatalog);
 testUtil.getAdmin().enableTable(systemCatalog);
+}catch(DoNotRetryIOException e){
+//table is not splittable
+assert(e.getMessage().contains("NOT splittable"));
+}
 
 // test again... Must still be exactly one region.
 rl = testUtil.getConnection().getRegionLocator(systemCatalog);

phoenix git commit: PHOENIX-4459 Region assignments are failing for the test cases with extended clocks to support SCN(Sergey Soldatov)

[ankit]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 eeea6c60f -> c489587e4


PHOENIX-4459 Region assignments are failing for the test cases with extended 
clocks to support SCN(Sergey Soldatov)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/c489587e
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/c489587e
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/c489587e

Branch: refs/heads/5.x-HBase-2.0
Commit: c489587e4ee91be922e7dca2ee7f1aba47d75a4d
Parents: eeea6c6
Author: Ankit Singhal 
Authored: Fri Jun 1 14:10:15 2018 -0700
Committer: Ankit Singhal 
Committed: Fri Jun 1 14:10:15 2018 -0700

--
 .../phoenix/end2end/ConcurrentMutationsIT.java  |  76 +-
 .../end2end/ExplainPlanWithStatsEnabledIT.java  |  17 ++-
 .../end2end/index/PartialIndexRebuilderIT.java  | 105 +++
 3 files changed, 54 insertions(+), 144 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/c489587e/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsIT.java
index 01133f1..ffc1049 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/ConcurrentMutationsIT.java
@@ -55,7 +55,6 @@ import org.junit.Test;
 import org.junit.runner.RunWith;
 
 @RunWith(RunUntilFailure.class)
-@Ignore
 public class ConcurrentMutationsIT extends ParallelStatsDisabledIT {
 private static final Random RAND = new Random(5);
 private static final String MVCC_LOCK_TEST_TABLE_PREFIX = "MVCCLOCKTEST_"; 
 
@@ -66,6 +65,7 @@ public class ConcurrentMutationsIT extends 
ParallelStatsDisabledIT {
 
 private static class MyClock extends EnvironmentEdge {
 public volatile long time;
+boolean shouldAdvance = true;
 
 public MyClock (long time) {
 this.time = time;
@@ -73,7 +73,14 @@ public class ConcurrentMutationsIT extends 
ParallelStatsDisabledIT {
 
 @Override
 public long currentTime() {
-return time;
+if (shouldAdvance) {
+return time++;
+} else {
+return time;
+}
+}
+public void setAdvance(boolean val) {
+shouldAdvance = val;
 }
 }
 
@@ -422,21 +429,15 @@ public class ConcurrentMutationsIT extends 
ParallelStatsDisabledIT {
 String tableName = generateUniqueName();
 String indexName = generateUniqueName();
 Properties props = 
PropertiesUtil.deepCopy(TestUtil.TEST_PROPERTIES);
-long ts = 1000;
-clock.time = ts;
-Connection conn = DriverManager.getConnection(getUrl(), props);
 
+Connection conn = DriverManager.getConnection(getUrl(), props);
 conn.createStatement().execute("CREATE TABLE " + tableName + "(k1 
CHAR(2) NOT NULL, k2 CHAR(2) NOT NULL, ts TIMESTAMP, V VARCHAR, V2 VARCHAR, 
CONSTRAINT pk PRIMARY KEY (k1,k2)) COLUMN_ENCODED_BYTES = 0");
 conn.close();
 
-ts = 1010;
-clock.time = ts;
 conn = DriverManager.getConnection(getUrl(), props);
 conn.createStatement().execute("CREATE INDEX " + indexName + " ON 
" + tableName + "(k2,k1,ts) INCLUDE (V, v2)");
 conn.close();
 
-ts = 1020;
-clock.time = ts;
-conn = DriverManager.getConnection(getUrl(), props);
+conn = DriverManager.getConnection(getUrl(), props);
 PreparedStatement stmt = conn.prepareStatement("UPSERT INTO " + 
tableName + " VALUES('aa','aa',?, '0')");
 stmt.setTimestamp(1, new Timestamp(1000L));
 stmt.executeUpdate();
@@ -444,8 +445,7 @@ public class ConcurrentMutationsIT extends 
ParallelStatsDisabledIT {
 conn.close();
 
 Timestamp expectedTimestamp;
-ts = 1040;
-clock.time = ts;
+clock.setAdvance(false);
 conn = DriverManager.getConnection(getUrl(), props);
 stmt = conn.prepareStatement("UPSERT INTO " + tableName + " 
VALUES('aa','aa',?, null)");
 expectedTimestamp = null;
@@ -455,10 +455,9 @@ public class ConcurrentMutationsIT extends 
ParallelStatsDisabledIT {
 stmt.setTimestamp(1, new Timestamp(3000L));
 stmt.executeUpdate();
 conn.commit();
+clock.setAdvance(true);
 conn.close();
 
-ts = 1050;
-clock.time = ts;
 conn = 

Build failed in Jenkins: Phoenix | Master #2041

[Apache Jenkins Server]

See 


Changes:

[jtaylor] PHOENIX-4762 Performance regression with transactional immutable 
indexes

--
[...truncated 101.54 KB...]
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.86 s 
- in org.apache.phoenix.end2end.UseSchemaIT
[INFO] Running org.apache.phoenix.end2end.index.ChildViewsUseParentViewIndexIT
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 406.283 
s - in org.apache.phoenix.end2end.UpgradeIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.108 s 
- in org.apache.phoenix.end2end.index.ChildViewsUseParentViewIndexIT
[INFO] Running org.apache.phoenix.end2end.index.DropMetadataIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.102 s 
- in org.apache.phoenix.end2end.index.DropMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.AsyncIndexDisabledIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.691 s 
- in org.apache.phoenix.end2end.index.AsyncIndexDisabledIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalImmutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.DropColumnIT
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 230.436 
s - in org.apache.phoenix.end2end.VariableLengthPKIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalIndexOptimizationIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.021 s 
- in org.apache.phoenix.end2end.index.GlobalIndexOptimizationIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalMutableNonTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 334.421 
s - in org.apache.phoenix.end2end.index.GlobalImmutableNonTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 358.884 
s - in org.apache.phoenix.end2end.index.GlobalImmutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 402.095 
s - in org.apache.phoenix.end2end.index.DropColumnIT
[INFO] Running org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.459 
s - in org.apache.phoenix.end2end.index.IndexMaintenanceIT
[INFO] Running org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 344.86 
s - in org.apache.phoenix.end2end.index.GlobalMutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 157.578 
s - in org.apache.phoenix.end2end.index.IndexMetadataIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 338.335 
s - in org.apache.phoenix.end2end.index.GlobalMutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 228.645 
s - in org.apache.phoenix.end2end.index.IndexWithTableSchemaChangeIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableNonTxIndexIT
[INFO] Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 364.491 
s - in org.apache.phoenix.end2end.index.IndexUsageIT
[INFO] Running org.apache.phoenix.end2end.index.LocalMutableTxIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 435.897 
s - in org.apache.phoenix.end2end.index.LocalImmutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.MutableIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 472.928 
s - in org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.SaltedIndexIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.794 s 
- in org.apache.phoenix.end2end.index.SaltedIndexIT
[INFO] Running org.apache.phoenix.end2end.index.ViewIndexIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 443.332 
s - in org.apache.phoenix.end2end.index.LocalMutableNonTxIndexIT
[INFO] Running org.apache.phoenix.end2end.index.txn.MutableRollbackIT
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 463.468 
s - in org.apache.phoenix.end2end.index.LocalMutableTxIndexIT
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 
143.701 s - in org.apache.phoenix.end2end.index.ViewIndexIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.278 s 
- in org.apache.phoenix.end2end.index.txn.MutableRollbackIT
[INFO] Running 

Apache-Phoenix | 4.x-HBase-1.2 | Build Successful

[Apache Jenkins Server]

4.x-HBase-1.2 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.2

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.2/lastCompletedBuild/testReport/

Changes
[jtaylor] PHOENIX-4762 Performance regression with transactional immutable indexes



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/5.x-HBase-2.0 d79c30023 -> eeea6c60f


http://git-wip-us.apache.org/repos/asf/phoenix/blob/eeea6c60/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 158f1e8..28dc87a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -398,5 +398,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public Table getTransactionalTableWriter(Table htable, PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTable transactionAwareHTable = new 
TransactionAwareHTable(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/eeea6c60/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index c6cbe3e..3d4c3ca 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -41,7 +41,6 @@ import org.apache.hadoop.hbase.HConstants;
 import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.Mutation;
@@ -116,7 +115,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -269,32 +267,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List indexMutations = 
Lists.newArrayListWithExpectedSize(dataMutations.size());
-for (final Mutation dataMutation : dataMutations) {
-long ts = MetaDataUtil.getClientTimeStamp(dataMutation);
-ptr.set(dataMutation.getRow());
-byte[] regionStartKey = null;
-byte[] regionEndkey = null;
-if(maintainer.isLocalIndex()) {
-HRegionLocation tableRegionLocation = 
connection.getQueryServices().getTableRegionLocation(table.getPhysicalName().getBytes(),
 dataMutation.getRow());
-regionStartKey = 
tableRegionLocation.getRegion().getStartKey();
-regionEndkey = tableRegionLocation.getRegion().getEndKey();
-}
-Delete delete = maintainer.buildDeleteMutation(kvBuilder, 
null, ptr, Collections.emptyList(), ts, regionStartKey, regionEndkey);
-
delete.setAttribute(PhoenixTransactionContext.TX_ROLLBACK_ATTRIBUTE_KEY, 
dataMutation.getAttribute(PhoenixTransactionContext.TX_ROLLBACK_ATTRIBUTE_KEY));
-indexMutations.add(delete);
-}
-return indexMutations;
-} catch (IOException e) {
-throw new SQLException(e);
-}
-}
-
 public static List generateIndexData(final PTable table, PTable 
index,
 final MultiRowMutationState multiRowMutationState, List 
dataMutations, final 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/eeea6c60
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/eeea6c60
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/eeea6c60

Branch: refs/heads/5.x-HBase-2.0
Commit: eeea6c60f299de73d0ebfd83b779f19476aa1fa9
Parents: d79c300
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 10:57:28 2018 -0700

--
 .../phoenix/end2end/index/MutableIndexIT.java   |   8 +-
 .../apache/phoenix/execute/MutationState.java   | 685 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 7 files changed, 379 insertions(+), 372 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/eeea6c60/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
index 87bc124..12e0dbf 100644
--- 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java
@@ -34,6 +34,8 @@ import java.util.Collection;
 import java.util.List;
 import java.util.Properties;
 
+import jline.internal.Log;
+
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.Admin;
 import org.apache.hadoop.hbase.client.Put;
@@ -70,8 +72,6 @@ import org.junit.runners.Parameterized.Parameters;
 
 import com.google.common.primitives.Doubles;
 
-import jline.internal.Log;
-
 @RunWith(Parameterized.class)
 public class MutableIndexIT extends ParallelStatsDisabledIT {
 
@@ -852,7 +852,7 @@ public class MutableIndexIT extends ParallelStatsDisabledIT 
{
   List regions = 
getUtility().getHBaseCluster().getRegions(hbaseTN);
   HRegion hRegion = regions.get(0);
   hRegion.flush(true);
-  HStore store = (HStore) hRegion.getStore(famBytes);
+  HStore store = hRegion.getStore(famBytes);
   store.triggerMajorCompaction();
   store.compactRecentForTestingAssumingDefaultPolicy(1);
 
@@ -860,7 +860,7 @@ public class MutableIndexIT extends ParallelStatsDisabledIT 
{
   regions = 
getUtility().getHBaseCluster().getRegions(TableName.valueOf(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME));
   hRegion = regions.get(0);
   hRegion.flush(true);
-  store = (HStore) 
hRegion.getStore(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES);
+  store = hRegion.getStore(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES);
   store.triggerMajorCompaction();
   store.compactRecentForTestingAssumingDefaultPolicy(1);
   }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/eeea6c60/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index 1d662ab..c3df314 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -83,7 +83,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -117,9 +116,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -151,30 +148,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize, maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
 this(maxSize, 

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-0.98 e6f9ce035 -> 5ce06a062


http://git-wip-us.apache.org/repos/asf/phoenix/blob/5ce06a06/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
index f3ad42f..751945a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
@@ -108,6 +108,11 @@ public interface PhoenixTransactionContext {
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable) {
 return null;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface 
htable, PTable table) {
+return null;
+}
 };
 /**
  * 
@@ -225,4 +230,5 @@ public interface PhoenixTransactionContext {
 public PhoenixTransactionContext 
newTransactionContext(PhoenixTransactionContext contex, boolean subTask);
 
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable);
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table);
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/5ce06a06/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 8b16210..bc33cff 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -397,5 +397,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTable transactionAwareHTable = new 
TransactionAwareHTable(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/5ce06a06/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index b4219ee..36a3c38 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -41,7 +41,6 @@ import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTableInterface;
@@ -115,7 +114,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -268,32 +266,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List indexMutations = 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/5ce06a06
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/5ce06a06
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/5ce06a06

Branch: refs/heads/4.x-HBase-0.98
Commit: 5ce06a062bb71c8846487ca35e657297fa3641f2
Parents: e6f9ce0
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 10:36:46 2018 -0700

--
 .../apache/phoenix/execute/MutationState.java   | 701 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 6 files changed, 385 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/5ce06a06/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index 74286bf..38b279c 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -38,10 +38,9 @@ import javax.annotation.Nonnull;
 import javax.annotation.concurrent.Immutable;
 
 import org.apache.hadoop.hbase.HConstants;
-import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.Pair;
@@ -81,7 +80,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -96,6 +94,7 @@ import org.apache.phoenix.transaction.TransactionFactory;
 import org.apache.phoenix.transaction.TransactionFactory.Provider;
 import org.apache.phoenix.util.ByteUtil;
 import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.phoenix.util.LogUtil;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.apache.phoenix.util.SQLCloseable;
@@ -116,9 +115,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -151,30 +148,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize, maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
-this(maxSize,maxSizeBytes, connection, false, txContext);
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
+this(maxSize, maxSizeBytes, connection, false, txContext);
 }
 
 public MutationState(MutationState mutationState) {
-this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState.getPhoenixTransactionContext());
+this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState
+.getPhoenixTransactionContext());
 }
 
 public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, long sizeOffset) {
 this(maxSize, maxSizeBytes, connection, false, null, sizeOffset);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext) {
+private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, subTask, txContext, 0);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext, long 
sizeOffset) {
-

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 9ccc32dd0 -> f8d09637c


http://git-wip-us.apache.org/repos/asf/phoenix/blob/f8d09637/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
index f3ad42f..751945a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
@@ -108,6 +108,11 @@ public interface PhoenixTransactionContext {
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable) {
 return null;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface 
htable, PTable table) {
+return null;
+}
 };
 /**
  * 
@@ -225,4 +230,5 @@ public interface PhoenixTransactionContext {
 public PhoenixTransactionContext 
newTransactionContext(PhoenixTransactionContext contex, boolean subTask);
 
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable);
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table);
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/f8d09637/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 7db54eb..58f5dc0 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -450,5 +450,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTableDelegate transactionAwareHTable = new 
TransactionAwareHTableDelegate(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/f8d09637/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index 7e280f4..78a68d2 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -42,7 +42,6 @@ import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTableInterface;
@@ -116,7 +115,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -269,32 +267,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/f8d09637
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/f8d09637
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/f8d09637

Branch: refs/heads/4.x-HBase-1.1
Commit: f8d09637caa9dc14c94e90b12c3916293a5c886a
Parents: 9ccc32d
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 10:34:21 2018 -0700

--
 .../apache/phoenix/execute/MutationState.java   | 701 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 6 files changed, 385 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/f8d09637/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index 774544b..a1f1446 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -38,10 +38,9 @@ import javax.annotation.Nonnull;
 import javax.annotation.concurrent.Immutable;
 
 import org.apache.hadoop.hbase.HConstants;
-import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.Pair;
@@ -83,7 +82,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -98,6 +96,7 @@ import org.apache.phoenix.transaction.TransactionFactory;
 import org.apache.phoenix.transaction.TransactionFactory.Provider;
 import org.apache.phoenix.util.ByteUtil;
 import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.phoenix.util.LogUtil;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.apache.phoenix.util.SQLCloseable;
@@ -116,9 +115,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -150,30 +147,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize,maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, false, txContext);
 }
 
 public MutationState(MutationState mutationState) {
-this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState.getPhoenixTransactionContext());
+this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState
+.getPhoenixTransactionContext());
 }
 
 public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, long sizeOffset) {
 this(maxSize, maxSizeBytes, connection, false, null, sizeOffset);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext) {
+private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, subTask, txContext, 0);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext, long 
sizeOffset) {
-this(maxSize, maxSizeBytes, connection, 

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-cdh5.11 5cc7575ef -> edb6dcf5f


http://git-wip-us.apache.org/repos/asf/phoenix/blob/edb6dcf5/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
index f3ad42f..751945a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
@@ -108,6 +108,11 @@ public interface PhoenixTransactionContext {
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable) {
 return null;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface 
htable, PTable table) {
+return null;
+}
 };
 /**
  * 
@@ -225,4 +230,5 @@ public interface PhoenixTransactionContext {
 public PhoenixTransactionContext 
newTransactionContext(PhoenixTransactionContext contex, boolean subTask);
 
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable);
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table);
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/edb6dcf5/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 9c6af63..94d6765 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -469,5 +469,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTableDelegate transactionAwareHTable = new 
TransactionAwareHTableDelegate(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/edb6dcf5/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index 7e280f4..78a68d2 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -42,7 +42,6 @@ import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTableInterface;
@@ -116,7 +115,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -269,32 +267,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/edb6dcf5
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/edb6dcf5
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/edb6dcf5

Branch: refs/heads/4.x-cdh5.11
Commit: edb6dcf5f976d91971941166746191a1b49be211
Parents: 5cc7575
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 10:30:40 2018 -0700

--
 .../apache/phoenix/execute/MutationState.java   | 701 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 6 files changed, 385 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/edb6dcf5/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index b9a6c9e..2e795b1 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -38,10 +38,9 @@ import javax.annotation.Nonnull;
 import javax.annotation.concurrent.Immutable;
 
 import org.apache.hadoop.hbase.HConstants;
-import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.Pair;
@@ -83,7 +82,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -98,6 +96,7 @@ import org.apache.phoenix.transaction.TransactionFactory;
 import org.apache.phoenix.transaction.TransactionFactory.Provider;
 import org.apache.phoenix.util.ByteUtil;
 import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.phoenix.util.LogUtil;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.apache.phoenix.util.SQLCloseable;
@@ -116,9 +115,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -150,30 +147,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize, maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, false, txContext);
 }
 
 public MutationState(MutationState mutationState) {
-this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState.getPhoenixTransactionContext());
+this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState
+.getPhoenixTransactionContext());
 }
 
 public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, long sizeOffset) {
 this(maxSize, maxSizeBytes, connection, false, null, sizeOffset);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext) {
+private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, subTask, txContext, 0);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext, long 
sizeOffset) {
-this(maxSize, maxSizeBytes, connection, 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/14baca9d
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/14baca9d
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/14baca9d

Branch: refs/heads/master
Commit: 14baca9ded1d68a97c1f559806fa34dfe8c9415e
Parents: 614c57d
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 10:18:49 2018 -0700

--
 .../apache/phoenix/execute/MutationState.java   | 701 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 6 files changed, 385 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/14baca9d/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index 52e490e..2e795b1 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -38,10 +38,9 @@ import javax.annotation.Nonnull;
 import javax.annotation.concurrent.Immutable;
 
 import org.apache.hadoop.hbase.HConstants;
-import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.Pair;
@@ -83,7 +82,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -98,6 +96,7 @@ import org.apache.phoenix.transaction.TransactionFactory;
 import org.apache.phoenix.transaction.TransactionFactory.Provider;
 import org.apache.phoenix.util.ByteUtil;
 import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.phoenix.util.LogUtil;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.apache.phoenix.util.SQLCloseable;
@@ -116,9 +115,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -150,30 +147,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize, maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, false, txContext);
 }
 
 public MutationState(MutationState mutationState) {
-this(mutationState.maxSize,  mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState.getPhoenixTransactionContext());
+this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState
+.getPhoenixTransactionContext());
 }
 
 public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, long sizeOffset) {
 this(maxSize, maxSizeBytes, connection, false, null, sizeOffset);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext) {
+private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, subTask, txContext, 0);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext, long 
sizeOffset) {
-this(maxSize, maxSizeBytes, connection, 

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/master 614c57d91 -> 14baca9de


http://git-wip-us.apache.org/repos/asf/phoenix/blob/14baca9d/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
index f3ad42f..751945a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
@@ -108,6 +108,11 @@ public interface PhoenixTransactionContext {
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable) {
 return null;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface 
htable, PTable table) {
+return null;
+}
 };
 /**
  * 
@@ -225,4 +230,5 @@ public interface PhoenixTransactionContext {
 public PhoenixTransactionContext 
newTransactionContext(PhoenixTransactionContext contex, boolean subTask);
 
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable);
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table);
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/14baca9d/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 8b16210..bc33cff 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -397,5 +397,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTable transactionAwareHTable = new 
TransactionAwareHTable(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/14baca9d/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index 7e280f4..78a68d2 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -42,7 +42,6 @@ import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTableInterface;
@@ -116,7 +115,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -269,32 +267,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List indexMutations = 

[2/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

PHOENIX-4762 Performance regression with transactional immutable indexes


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/f3119320
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/f3119320
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/f3119320

Branch: refs/heads/4.x-HBase-1.3
Commit: f3119320b6d263252a790c6a5cd96dce5a714ed6
Parents: 24af104
Author: James Taylor 
Authored: Fri Jun 1 09:03:21 2018 -0700
Committer: James Taylor 
Committed: Fri Jun 1 09:03:21 2018 -0700

--
 .../apache/phoenix/execute/MutationState.java   | 701 ++-
 .../PhoenixTxIndexMutationGenerator.java|   2 +-
 .../transaction/OmidTransactionContext.java |   6 +
 .../transaction/PhoenixTransactionContext.java  |   6 +
 .../transaction/TephraTransactionContext.java   |  16 +
 .../java/org/apache/phoenix/util/IndexUtil.java |  28 -
 6 files changed, 385 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/f3119320/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java 
b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
index 52e490e..2e795b1 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
@@ -38,10 +38,9 @@ import javax.annotation.Nonnull;
 import javax.annotation.concurrent.Immutable;
 
 import org.apache.hadoop.hbase.HConstants;
-import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Mutation;
 import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.Pair;
@@ -83,7 +82,6 @@ import org.apache.phoenix.schema.PMetaData;
 import org.apache.phoenix.schema.PName;
 import org.apache.phoenix.schema.PRow;
 import org.apache.phoenix.schema.PTable;
-import org.apache.phoenix.schema.PTable.IndexType;
 import org.apache.phoenix.schema.PTableRef;
 import org.apache.phoenix.schema.PTableType;
 import org.apache.phoenix.schema.RowKeySchema;
@@ -98,6 +96,7 @@ import org.apache.phoenix.transaction.TransactionFactory;
 import org.apache.phoenix.transaction.TransactionFactory.Provider;
 import org.apache.phoenix.util.ByteUtil;
 import org.apache.phoenix.util.IndexUtil;
+import org.apache.phoenix.util.KeyValueUtil;
 import org.apache.phoenix.util.LogUtil;
 import org.apache.phoenix.util.PhoenixRuntime;
 import org.apache.phoenix.util.SQLCloseable;
@@ -116,9 +115,7 @@ import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 
 /**
- * 
  * Tracks the uncommitted state
- *
  */
 public class MutationState implements SQLCloseable {
 private static final Logger logger = 
LoggerFactory.getLogger(MutationState.class);
@@ -150,30 +147,34 @@ public class MutationState implements SQLCloseable {
 this(maxSize, maxSizeBytes, connection, false, null);
 }
 
-public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, PhoenixTransactionContext txContext) {
+public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, false, txContext);
 }
 
 public MutationState(MutationState mutationState) {
-this(mutationState.maxSize,  mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState.getPhoenixTransactionContext());
+this(mutationState.maxSize, mutationState.maxSizeBytes, 
mutationState.connection, true, mutationState
+.getPhoenixTransactionContext());
 }
 
 public MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, long sizeOffset) {
 this(maxSize, maxSizeBytes, connection, false, null, sizeOffset);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext) {
+private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask,
+PhoenixTransactionContext txContext) {
 this(maxSize, maxSizeBytes, connection, subTask, txContext, 0);
 }
 
-private MutationState(long maxSize, long maxSizeBytes, PhoenixConnection 
connection, boolean subTask, PhoenixTransactionContext txContext, long 
sizeOffset) {
-this(maxSize, maxSizeBytes, connection, 

[1/2] phoenix git commit: PHOENIX-4762 Performance regression with transactional immutable indexes

[jamestaylor]

Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.3 24af10405 -> f3119320b


http://git-wip-us.apache.org/repos/asf/phoenix/blob/f3119320/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
index f3ad42f..751945a 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/PhoenixTransactionContext.java
@@ -108,6 +108,11 @@ public interface PhoenixTransactionContext {
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable) {
 return null;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface 
htable, PTable table) {
+return null;
+}
 };
 /**
  * 
@@ -225,4 +230,5 @@ public interface PhoenixTransactionContext {
 public PhoenixTransactionContext 
newTransactionContext(PhoenixTransactionContext contex, boolean subTask);
 
 public HTableInterface getTransactionalTable(HTableInterface htable, 
boolean isImmutable);
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table);
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/f3119320/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
index 8b16210..bc33cff 100644
--- 
a/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/transaction/TephraTransactionContext.java
@@ -397,5 +397,21 @@ public class TephraTransactionContext implements 
PhoenixTransactionContext {
 this.addTransactionAware(transactionAwareHTable);
 return transactionAwareHTable;
 }
+
+@Override
+public HTableInterface getTransactionalTableWriter(HTableInterface htable, 
PTable table) {
+boolean isIndex = table.getType() == PTableType.INDEX;
+TransactionAwareHTable transactionAwareHTable = new 
TransactionAwareHTable(htable, table.isImmutableRows() || isIndex ? 
TxConstants.ConflictDetection.NONE : TxConstants.ConflictDetection.ROW);
+// Don't add immutable indexes (those are the only ones that would 
participate
+// during a commit), as we don't need conflict detection for these.
+if (isIndex) {
+transactionAwareHTable.startTx(getTransaction());
+} else {
+// Even for immutable, we need to do this so that an abort has the 
state
+// necessary to generate the rows to delete.
+this.addTransactionAware(transactionAwareHTable);
+}
+return transactionAwareHTable;
+}
 
 }

http://git-wip-us.apache.org/repos/asf/phoenix/blob/f3119320/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
--
diff --git a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java 
b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
index 7e280f4..78a68d2 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/util/IndexUtil.java
@@ -42,7 +42,6 @@ import org.apache.hadoop.hbase.HRegionLocation;
 import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.Delete;
 import org.apache.hadoop.hbase.client.Durability;
 import org.apache.hadoop.hbase.client.Get;
 import org.apache.hadoop.hbase.client.HTableInterface;
@@ -116,7 +115,6 @@ import org.apache.phoenix.schema.types.PDecimal;
 import org.apache.phoenix.schema.types.PLong;
 import org.apache.phoenix.schema.types.PVarbinary;
 import org.apache.phoenix.schema.types.PVarchar;
-import org.apache.phoenix.transaction.PhoenixTransactionContext;
 
 import com.google.common.collect.Lists;
 
@@ -269,32 +267,6 @@ public class IndexUtil {
 .getLength()) == 0);
 }
 
-public static List generateDeleteIndexData(final PTable table, 
PTable index,
-List dataMutations, ImmutableBytesWritable ptr, final 
KeyValueBuilder kvBuilder, PhoenixConnection connection)
-throws SQLException {
-try {
-IndexMaintainer maintainer = index.getIndexMaintainer(table, 
connection);
-List indexMutations = 

Build failed in Jenkins: Phoenix Compile Compatibility with HBase #654

[Apache Jenkins Server]

See 


--
[...truncated 37.23 KB...]
  symbol:   class HBaseRpcController
  location: class 
org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory
[ERROR] 
:[52,9]
 cannot find symbol
  symbol:   class HBaseRpcController
  location: class 
org.apache.hadoop.hbase.ipc.controller.ClientRpcControllerFactory
[ERROR] 
:[180,14]
 cannot find symbol
  symbol: class MetricRegistry
[ERROR] 
:[179,7]
 method does not override or implement a method from a supertype
[ERROR] 
:[454,78]
 cannot find symbol
  symbol: class HBaseRpcController
[ERROR] 
:[432,17]
 cannot find symbol
  symbol: class HBaseRpcController
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on 
project phoenix-core: Compilation failure: Compilation failure: 
[ERROR] 
:[34,39]
 cannot find symbol
[ERROR]   symbol:   class MetricRegistry
[ERROR]   location: package org.apache.hadoop.hbase.metrics
[ERROR] 
:[144,16]
 cannot find symbol
[ERROR]   symbol:   class MetricRegistry
[ERROR]   location: class 
org.apache.phoenix.coprocessor.PhoenixMetaDataCoprocessorHost.PhoenixMetaDataControllerEnvironment
[ERROR] 
:[24,35]
 cannot find symbol
[ERROR]   symbol:   class DelegatingHBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[25,35]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[37,37]
 cannot find symbol
[ERROR]   symbol: class DelegatingHBaseRpcController
[ERROR] 
:[56,38]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.MetadataRpcController
[ERROR] 
:[26,35]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: package org.apache.hadoop.hbase.ipc
[ERROR] 
:[40,12]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.InterRegionServerMetadataRpcControllerFactory
[ERROR] 
:[46,12]
 cannot find symbol
[ERROR]   symbol:   class HBaseRpcController
[ERROR]   location: class 
org.apache.hadoop.hbase.ipc.controller.InterRegionServerMetadataRpcControllerFactory
[ERROR]