Apache-Phoenix | 4.x-HBase-1.1 | Build Successful

2017-08-28 Thread Apache Jenkins Server 
4.x-HBase-1.1 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.1

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastCompletedBuild/testReport/

Changes
[jtaylor] PHOENIX-2460 Implement scrutiny command to validate whether  or not an



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

Build failed in Jenkins: Phoenix | Master #1754

2017-08-28 Thread Apache Jenkins Server 
See 


Changes:

[jtaylor] PHOENIX-2460 Implement scrutiny command to validate whether  or not an

--
[...truncated 130.79 KB...]
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.phoenix.end2end.DistinctCountIT
[INFO] Running org.apache.phoenix.end2end.CustomEntityDataIT
[INFO] Running org.apache.phoenix.end2end.ExtendedQueryExecIT
[INFO] Running org.apache.phoenix.end2end.DropSchemaIT
[INFO] Running org.apache.phoenix.end2end.FunkyNamesIT
[INFO] Running org.apache.phoenix.end2end.DerivedTableIT
[INFO] Running org.apache.phoenix.end2end.CreateTableIT
[INFO] Running org.apache.phoenix.end2end.CreateSchemaIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.929 s 
- in org.apache.phoenix.end2end.CreateSchemaIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.962 s 
- in org.apache.phoenix.end2end.FunkyNamesIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.265 s 
- in org.apache.phoenix.end2end.CustomEntityDataIT
[INFO] Running org.apache.phoenix.end2end.ProductMetricsIT
[INFO] Running org.apache.phoenix.end2end.QueryDatabaseMetaDataIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.657 s 
- in org.apache.phoenix.end2end.ExtendedQueryExecIT
[INFO] Running org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Running org.apache.phoenix.end2end.NativeHBaseTypesIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.417 s 
- in org.apache.phoenix.end2end.DropSchemaIT
[INFO] Running org.apache.phoenix.end2end.RowValueConstructorIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.049 s 
- in org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.065 s 
- in org.apache.phoenix.end2end.NativeHBaseTypesIT
[INFO] Running org.apache.phoenix.end2end.SequenceBulkAllocationIT
[INFO] Running org.apache.phoenix.end2end.SequenceIT
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.484 
s - in org.apache.phoenix.end2end.DistinctCountIT
[INFO] Running org.apache.phoenix.end2end.ToNumberFunctionIT
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.889 
s - in org.apache.phoenix.end2end.DerivedTableIT
[INFO] Running org.apache.phoenix.end2end.TopNIT
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.027 
s - in org.apache.phoenix.end2end.ToNumberFunctionIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.259 s 
- in org.apache.phoenix.end2end.TopNIT
[INFO] Tests run: 61, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.424 
s - in org.apache.phoenix.end2end.ProductMetricsIT
[INFO] Running org.apache.phoenix.end2end.TruncateFunctionIT
[INFO] Running org.apache.phoenix.end2end.UpsertValuesIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.417 s 
- in org.apache.phoenix.end2end.TruncateFunctionIT
[INFO] Running org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.447 
s - in org.apache.phoenix.end2end.SequenceIT
[INFO] Tests run: 56, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.462 
s - in org.apache.phoenix.end2end.SequenceBulkAllocationIT
[INFO] Running org.apache.phoenix.end2end.VariableLengthPKIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.399 s 
- in org.apache.phoenix.end2end.salted.SaltedTableIT
[INFO] Running org.apache.phoenix.rpc.UpdateCacheWithScnIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.865 s 
- in org.apache.phoenix.rpc.UpdateCacheWithScnIT
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.754 
s - in org.apache.phoenix.end2end.VariableLengthPKIT
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 156.259 
s - in org.apache.phoenix.end2end.QueryDatabaseMetaDataIT
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 146.863 
s - in org.apache.phoenix.end2end.RowValueConstructorIT
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 209.762 
s - in org.apache.phoenix.end2end.CreateTableIT
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.589 
s - in org.apache.phoenix.end2end.UpsertValuesIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 414, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:integration-test (HBaseManagedTimeTests) 
@ phoenix-core ---
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0,

Build failed in Jenkins: Phoenix | Master #1753

2017-08-28 Thread Apache Jenkins Server 
See 


Changes:

[jtaylor] PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)

--
[...truncated 80.60 KB...]
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.713 
s - in org.apache.phoenix.end2end.DefaultColumnValueIT
[INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 461.745 
s - in org.apache.phoenix.end2end.AlterTableWithViewsIT
[INFO] Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 151.395 
s - in org.apache.phoenix.end2end.DateTimeIT
[INFO] Running org.apache.phoenix.end2end.DeleteIT
[INFO] Running org.apache.phoenix.end2end.DisableLocalIndexIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.184 s 
- in org.apache.phoenix.end2end.DisableLocalIndexIT
[INFO] Running org.apache.phoenix.end2end.DynamicColumnIT
[INFO] Running org.apache.phoenix.end2end.DistinctPrefixFilterIT
[INFO] Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 311.159 
s - in org.apache.phoenix.end2end.CastAndCoerceIT
[INFO] Tests run: 56, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 343.854 
s - in org.apache.phoenix.end2end.CaseStatementIT
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.722 s 
- in org.apache.phoenix.end2end.DynamicColumnIT
[INFO] Running org.apache.phoenix.end2end.DynamicUpsertIT
[INFO] Running org.apache.phoenix.end2end.DynamicFamilyIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.601 s 
- in org.apache.phoenix.end2end.DynamicUpsertIT
[INFO] Running org.apache.phoenix.end2end.EvaluationOfORIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.405 s 
- in org.apache.phoenix.end2end.EvaluationOfORIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.052 s 
- in org.apache.phoenix.end2end.DynamicFamilyIT
[INFO] Running org.apache.phoenix.end2end.ExecuteStatementsIT
[INFO] Running org.apache.phoenix.end2end.ExpFunctionEnd2EndIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.503 s 
- in org.apache.phoenix.end2end.ExpFunctionEnd2EndIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.641 s 
- in org.apache.phoenix.end2end.ExecuteStatementsIT
[INFO] Running org.apache.phoenix.end2end.FirstValueFunctionIT
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 108.904 
s - in org.apache.phoenix.end2end.DeleteIT
[INFO] Running org.apache.phoenix.end2end.ExplainPlanWithStatsDisabledIT
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 s 
- in org.apache.phoenix.end2end.ExplainPlanWithStatsDisabledIT
[INFO] Running org.apache.phoenix.end2end.EncodeFunctionIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.495 s 
- in org.apache.phoenix.end2end.FirstValueFunctionIT
[INFO] Running org.apache.phoenix.end2end.GetSetByteBitFunctionEnd2EndIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.415 s 
- in org.apache.phoenix.end2end.GetSetByteBitFunctionEnd2EndIT
[INFO] Running org.apache.phoenix.end2end.GroupByIT
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.523 s 
- in org.apache.phoenix.end2end.EncodeFunctionIT
[INFO] Running org.apache.phoenix.end2end.HashJoinIT
[INFO] Running org.apache.phoenix.end2end.FlappingAlterTableIT
[INFO] Running org.apache.phoenix.end2end.FirstValuesFunctionIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.43 s - 
in org.apache.phoenix.end2end.FirstValuesFunctionIT
[INFO] Running org.apache.phoenix.end2end.HashJoinLocalIndexIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.891 s 
- in org.apache.phoenix.end2end.HashJoinLocalIndexIT
[INFO] Running org.apache.phoenix.end2end.HashJoinMoreIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.031 s 
- in org.apache.phoenix.end2end.FlappingAlterTableIT
[INFO] Running org.apache.phoenix.end2end.ImmutableTablePropertiesIT
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 131.267 
s - in org.apache.phoenix.end2end.DistinctPrefixFilterIT
[INFO] Running org.apache.phoenix.end2end.InListIT
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.837 s 
- in org.apache.phoenix.end2end.ImmutableTablePropertiesIT
[INFO] Running org.apache.phoenix.end2end.InstrFunctionIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.764 s 
- in org.apache.phoenix.end2end.InstrFunctionIT
[INFO] Running org.apache.phoenix.end2end.IsNullIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.124 s 
- in org.apache.phoenix.end2end.IsNullIT
[INFO] Running org.apache.phoenix.end2end.LastValueFunctionIT
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.121 s 
- in org.apache.phoenix.end2end.HashJoinMoreIT
[INFO]

Apache-Phoenix | 4.x-HBase-1.1 | Build Successful

2017-08-28 Thread Apache Jenkins Server 
4.x-HBase-1.1 branch build status Successful

Source repository https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=shortlog;h=refs/heads/4.x-HBase-1.1

Compiled Artifacts https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastSuccessfulBuild/artifact/

Test Report https://builds.apache.org/job/Phoenix-4.x-HBase-1.1/lastCompletedBuild/testReport/

Changes
[jtaylor] PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)



Build times for last couple of runsLatest build time is the right most | Legend blue: normal, red: test failure, gray: timeout

[2/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
PHOENIX-2460 Implement scrutiny command to validate whether
 or not an index is in sync with the data table (Vincent Poon)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/549d22da
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/549d22da
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/549d22da

Branch: refs/heads/4.x-HBase-1.2
Commit: 549d22da904ef25113b714425271e1ab1f26fdf4
Parents: 4a81d92
Author: James Taylor 
Authored: Mon Aug 28 15:32:20 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:43:30 2017 -0700

--
 .../phoenix/end2end/IndexScrutinyToolIT.java| 612 +++
 .../mapreduce/index/IndexScrutinyMapper.java| 377 
 .../index/IndexScrutinyTableOutput.java | 345 +++
 .../mapreduce/index/IndexScrutinyTool.java  | 522 
 .../mapreduce/index/PhoenixIndexDBWritable.java |  11 +-
 .../index/PhoenixScrutinyJobCounters.java   |  41 ++
 .../index/SourceTargetColumnNames.java  | 195 ++
 .../mapreduce/util/IndexColumnNames.java| 239 
 .../util/PhoenixConfigurationUtil.java  | 128 +++-
 .../java/org/apache/phoenix/util/QueryUtil.java |  63 +-
 .../org/apache/phoenix/util/SchemaUtil.java |  21 +
 .../phoenix/mapreduce/index/BaseIndexTest.java  |  97 +++
 .../index/IndexScrutinyTableOutputTest.java |  87 +++
 .../mapreduce/util/IndexColumnNamesTest.java|  74 +++
 .../org/apache/phoenix/util/QueryUtilTest.java  |  18 +-
 15 files changed, 2817 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/549d22da/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
new file mode 100644
index 000..5068610
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
@@ -0,0 +1,612 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless 
required by applicable
+ * law or agreed to in writing, software distributed under the License is 
distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied. See the License
+ * for the specific language governing permissions and limitations under the 
License.
+ */
+package org.apache.phoenix.end2end;
+
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BAD_COVERED_COL_VAL_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BATCHES_PROCESSED_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.INVALID_ROW_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.VALID_ROW_COUNT;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.UUID;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.mapreduce.Counters;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTableOutput;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.OutputFormat;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.SourceTable;
+import org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters;
+import org.apache.phoenix.mapreduce.index.SourceTargetColumnNames;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;

[1/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 4a81d9277 -> 549d22da9


http://git-wip-us.apache.org/repos/asf/phoenix/blob/549d22da/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
new file mode 100644
index 000..f3ff39e
--- /dev/null
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.mapreduce.index;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.util.ConnectionUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixMapReduceUtil;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * An MR job to verify that the index table is in sync with the data table.
+ *
+ */
+public class IndexScrutinyTool extends Configured implements Tool {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(IndexScrutinyTool.class);
+
+private static final Option SCHEMA_NAME_OPTION =
+new Option("s", "schema", true, "Phoenix schema name (optional)");
+private static final Option DATA_TABLE_OPTION =
+new Option("dt", "data-table", true, "Data table name 
(mandatory)");
+private static final Option INDEX_TABLE_OPTION =
+new Option("it", "index-table", true,
+"Index table name (mandatory).");
+private static final Option TIMESTAMP =
+new Option("t", "time", true,
+"Timestamp in millis used to compare the index and data 
tables.  Defaults to current time minus 60 seconds");
+
+private static final Option RUN_FOREGROUND_OPTION =
+new Option("runfg", "run-foreground", false, "Applicable on top of 
-direct option."
++ "If specified, runs index scrutiny in Foreground. 
Default - Runs the build in background.");
+
+private static final Option SNAPSHOT_OPTION = //TODO check if this works
+new Option("snap", "snapshot", false,
+"If specified, uses Snapshots for

[2/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
PHOENIX-2460 Implement scrutiny command to validate whether
 or not an index is in sync with the data table (Vincent Poon)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/124404ae
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/124404ae
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/124404ae

Branch: refs/heads/4.x-HBase-1.1
Commit: 124404ae2db6f49a82d92d5f07010c6390b16746
Parents: cce7ab2
Author: James Taylor 
Authored: Mon Aug 28 15:32:20 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:42:13 2017 -0700

--
 .../phoenix/end2end/IndexScrutinyToolIT.java| 612 +++
 .../mapreduce/index/IndexScrutinyMapper.java| 377 
 .../index/IndexScrutinyTableOutput.java | 345 +++
 .../mapreduce/index/IndexScrutinyTool.java  | 522 
 .../mapreduce/index/PhoenixIndexDBWritable.java |  11 +-
 .../index/PhoenixScrutinyJobCounters.java   |  41 ++
 .../index/SourceTargetColumnNames.java  | 195 ++
 .../mapreduce/util/IndexColumnNames.java| 239 
 .../util/PhoenixConfigurationUtil.java  | 128 +++-
 .../java/org/apache/phoenix/util/QueryUtil.java |  63 +-
 .../org/apache/phoenix/util/SchemaUtil.java |  21 +
 .../phoenix/mapreduce/index/BaseIndexTest.java  |  97 +++
 .../index/IndexScrutinyTableOutputTest.java |  87 +++
 .../mapreduce/util/IndexColumnNamesTest.java|  74 +++
 .../org/apache/phoenix/util/QueryUtilTest.java  |  18 +-
 15 files changed, 2817 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/124404ae/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
new file mode 100644
index 000..5068610
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
@@ -0,0 +1,612 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless 
required by applicable
+ * law or agreed to in writing, software distributed under the License is 
distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied. See the License
+ * for the specific language governing permissions and limitations under the 
License.
+ */
+package org.apache.phoenix.end2end;
+
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BAD_COVERED_COL_VAL_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BATCHES_PROCESSED_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.INVALID_ROW_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.VALID_ROW_COUNT;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.UUID;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.mapreduce.Counters;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTableOutput;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.OutputFormat;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.SourceTable;
+import org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters;
+import org.apache.phoenix.mapreduce.index.SourceTargetColumnNames;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;

[1/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.1 cce7ab29a -> 124404ae2


http://git-wip-us.apache.org/repos/asf/phoenix/blob/124404ae/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
new file mode 100644
index 000..f3ff39e
--- /dev/null
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.mapreduce.index;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.util.ConnectionUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixMapReduceUtil;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * An MR job to verify that the index table is in sync with the data table.
+ *
+ */
+public class IndexScrutinyTool extends Configured implements Tool {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(IndexScrutinyTool.class);
+
+private static final Option SCHEMA_NAME_OPTION =
+new Option("s", "schema", true, "Phoenix schema name (optional)");
+private static final Option DATA_TABLE_OPTION =
+new Option("dt", "data-table", true, "Data table name 
(mandatory)");
+private static final Option INDEX_TABLE_OPTION =
+new Option("it", "index-table", true,
+"Index table name (mandatory).");
+private static final Option TIMESTAMP =
+new Option("t", "time", true,
+"Timestamp in millis used to compare the index and data 
tables.  Defaults to current time minus 60 seconds");
+
+private static final Option RUN_FOREGROUND_OPTION =
+new Option("runfg", "run-foreground", false, "Applicable on top of 
-direct option."
++ "If specified, runs index scrutiny in Foreground. 
Default - Runs the build in background.");
+
+private static final Option SNAPSHOT_OPTION = //TODO check if this works
+new Option("snap", "snapshot", false,
+"If specified, uses Snapshots for

[1/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-0.98 ce6810f42 -> 08e2a29f6


http://git-wip-us.apache.org/repos/asf/phoenix/blob/08e2a29f/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
new file mode 100644
index 000..f3ff39e
--- /dev/null
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.mapreduce.index;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.util.ConnectionUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixMapReduceUtil;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * An MR job to verify that the index table is in sync with the data table.
+ *
+ */
+public class IndexScrutinyTool extends Configured implements Tool {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(IndexScrutinyTool.class);
+
+private static final Option SCHEMA_NAME_OPTION =
+new Option("s", "schema", true, "Phoenix schema name (optional)");
+private static final Option DATA_TABLE_OPTION =
+new Option("dt", "data-table", true, "Data table name 
(mandatory)");
+private static final Option INDEX_TABLE_OPTION =
+new Option("it", "index-table", true,
+"Index table name (mandatory).");
+private static final Option TIMESTAMP =
+new Option("t", "time", true,
+"Timestamp in millis used to compare the index and data 
tables.  Defaults to current time minus 60 seconds");
+
+private static final Option RUN_FOREGROUND_OPTION =
+new Option("runfg", "run-foreground", false, "Applicable on top of 
-direct option."
++ "If specified, runs index scrutiny in Foreground. 
Default - Runs the build in background.");
+
+private static final Option SNAPSHOT_OPTION = //TODO check if this works
+new Option("snap", "snapshot", false,
+"If specified, uses Snapshots for

[2/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
PHOENIX-2460 Implement scrutiny command to validate whether
 or not an index is in sync with the data table (Vincent Poon)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/08e2a29f
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/08e2a29f
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/08e2a29f

Branch: refs/heads/4.x-HBase-0.98
Commit: 08e2a29f60bcedd107f96fc50cfeb8996296d56f
Parents: ce6810f
Author: James Taylor 
Authored: Mon Aug 28 15:32:20 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:40:56 2017 -0700

--
 .../phoenix/end2end/IndexScrutinyToolIT.java| 612 +++
 .../mapreduce/index/IndexScrutinyMapper.java| 377 
 .../index/IndexScrutinyTableOutput.java | 345 +++
 .../mapreduce/index/IndexScrutinyTool.java  | 522 
 .../mapreduce/index/PhoenixIndexDBWritable.java |  11 +-
 .../index/PhoenixScrutinyJobCounters.java   |  41 ++
 .../index/SourceTargetColumnNames.java  | 195 ++
 .../mapreduce/util/IndexColumnNames.java| 239 
 .../util/PhoenixConfigurationUtil.java  | 128 +++-
 .../java/org/apache/phoenix/util/QueryUtil.java |  63 +-
 .../org/apache/phoenix/util/SchemaUtil.java |  21 +
 .../phoenix/mapreduce/index/BaseIndexTest.java  |  97 +++
 .../index/IndexScrutinyTableOutputTest.java |  87 +++
 .../mapreduce/util/IndexColumnNamesTest.java|  74 +++
 .../org/apache/phoenix/util/QueryUtilTest.java  |  18 +-
 15 files changed, 2817 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/08e2a29f/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
new file mode 100644
index 000..5068610
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexScrutinyToolIT.java
@@ -0,0 +1,612 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless 
required by applicable
+ * law or agreed to in writing, software distributed under the License is 
distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied. See the License
+ * for the specific language governing permissions and limitations under the 
License.
+ */
+package org.apache.phoenix.end2end;
+
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BAD_COVERED_COL_VAL_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.BATCHES_PROCESSED_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.INVALID_ROW_COUNT;
+import static 
org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters.VALID_ROW_COUNT;
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.UUID;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.mapreduce.Counters;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTableOutput;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.OutputFormat;
+import org.apache.phoenix.mapreduce.index.IndexScrutinyTool.SourceTable;
+import org.apache.phoenix.mapreduce.index.PhoenixScrutinyJobCounters;
+import org.apache.phoenix.mapreduce.index.SourceTargetColumnNames;
+import

[1/2] phoenix git commit: PHOENIX-2460 Implement scrutiny command to validate whether or not an index is in sync with the data table (Vincent Poon)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/master d6381afc3 -> fc6594883


http://git-wip-us.apache.org/repos/asf/phoenix/blob/fc659488/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
--
diff --git 
a/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
new file mode 100644
index 000..f3ff39e
--- /dev/null
+++ 
b/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexScrutinyTool.java
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.mapreduce.index;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.cli.PosixParser;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.phoenix.jdbc.PhoenixConnection;
+import org.apache.phoenix.mapreduce.CsvBulkImportUtil;
+import org.apache.phoenix.mapreduce.util.ConnectionUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
+import org.apache.phoenix.mapreduce.util.PhoenixMapReduceUtil;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.util.EnvironmentEdgeManager;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.phoenix.util.QueryUtil;
+import org.apache.phoenix.util.SchemaUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+/**
+ * An MR job to verify that the index table is in sync with the data table.
+ *
+ */
+public class IndexScrutinyTool extends Configured implements Tool {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(IndexScrutinyTool.class);
+
+private static final Option SCHEMA_NAME_OPTION =
+new Option("s", "schema", true, "Phoenix schema name (optional)");
+private static final Option DATA_TABLE_OPTION =
+new Option("dt", "data-table", true, "Data table name 
(mandatory)");
+private static final Option INDEX_TABLE_OPTION =
+new Option("it", "index-table", true,
+"Index table name (mandatory).");
+private static final Option TIMESTAMP =
+new Option("t", "time", true,
+"Timestamp in millis used to compare the index and data 
tables.  Defaults to current time minus 60 seconds");
+
+private static final Option RUN_FOREGROUND_OPTION =
+new Option("runfg", "run-foreground", false, "Applicable on top of 
-direct option."
++ "If specified, runs index scrutiny in Foreground. 
Default - Runs the build in background.");
+
+private static final Option SNAPSHOT_OPTION = //TODO check if this works
+new Option("snap", "snapshot", false,
+"If specified, uses Snapshots for async index

phoenix git commit: PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-1.2 3f4e72324 -> 4a81d9277


PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/4a81d927
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/4a81d927
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/4a81d927

Branch: refs/heads/4.x-HBase-1.2
Commit: 4a81d9277ba93dc9d3969ac416ed2215ef73810d
Parents: 3f4e723
Author: James Taylor 
Authored: Mon Aug 28 15:17:37 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:22:04 2017 -0700

--
 dev/release_files/NOTICE|   8 +
 phoenix-core/pom.xml|   5 +
 .../CountDistinctApproximateHyperLogLogIT.java  | 154 +++
 .../phoenix/expression/ExpressionType.java  |   4 +-
 ...stinctCountHyperLogLogAggregateFunction.java | 192 +++
 ...tinctCountHyperLogLogAggregateParseNode.java |  39 
 6 files changed, 401 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/4a81d927/dev/release_files/NOTICE
--
diff --git a/dev/release_files/NOTICE b/dev/release_files/NOTICE
index eed1edc..b7b40c0 100644
--- a/dev/release_files/NOTICE
+++ b/dev/release_files/NOTICE
@@ -277,3 +277,11 @@ jcip-annotations
 findbugs-annotations
 
   Copyright Stephen Connolly.
+
+stream-lib
+
+  Copyright 2016 AddThis.
+  This product includes software developed by AddThis.
+  This product also includes code adapted from:
+- software copyright (c) 2014 The Apache Software Foundation., 
http://lucene.apache.org/solr/
+- software copyright (c) 2014 The Apache Software Foundation., 
http://mahout.apache.org/

http://git-wip-us.apache.org/repos/asf/phoenix/blob/4a81d927/phoenix-core/pom.xml
--
diff --git a/phoenix-core/pom.xml b/phoenix-core/pom.xml
index 275b72f..ce44171 100644
--- a/phoenix-core/pom.xml
+++ b/phoenix-core/pom.xml
@@ -470,5 +470,10 @@
 joni
 ${joni.version}
 
+
+  com.clearspring.analytics
+  stream
+  2.9.5
+
   
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/4a81d927/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
new file mode 100644
index 000..3de1509
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.*;
+import java.util.Properties;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.*;
+
+public class CountDistinctApproximateHyperLogLogIT extends 
ParallelStatsDisabledIT {
+   private String tableName;
+
+   @Before
+   public void generateTableNames() {
+   tableName = "T_" + generateUniqueName();
+   }
+
+   @Test(expected = ColumnNotFoundException.class)
+   public void testDistinctCountException() throws Exception {
+   String query = "SELECT APPROX_COUNT_DISTINCT(x) FROM " + 
tableName;
+   Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+   
+   try (Connection conn = DriverManager.getConnection(getUrl(), 
props);
+   PreparedStatement statement =

phoenix git commit: PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/4.x-HBase-0.98 793f26a4b -> ce6810f42


PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/ce6810f4
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/ce6810f4
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/ce6810f4

Branch: refs/heads/4.x-HBase-0.98
Commit: ce6810f42310a368183ed5c9185ab1e16f83dbd0
Parents: 793f26a
Author: James Taylor 
Authored: Mon Aug 28 15:17:37 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:19:12 2017 -0700

--
 dev/release_files/NOTICE|   8 +
 phoenix-core/pom.xml|   5 +
 .../CountDistinctApproximateHyperLogLogIT.java  | 154 +++
 .../phoenix/expression/ExpressionType.java  |   4 +-
 ...stinctCountHyperLogLogAggregateFunction.java | 192 +++
 ...tinctCountHyperLogLogAggregateParseNode.java |  39 
 6 files changed, 401 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/ce6810f4/dev/release_files/NOTICE
--
diff --git a/dev/release_files/NOTICE b/dev/release_files/NOTICE
index eed1edc..b7b40c0 100644
--- a/dev/release_files/NOTICE
+++ b/dev/release_files/NOTICE
@@ -277,3 +277,11 @@ jcip-annotations
 findbugs-annotations
 
   Copyright Stephen Connolly.
+
+stream-lib
+
+  Copyright 2016 AddThis.
+  This product includes software developed by AddThis.
+  This product also includes code adapted from:
+- software copyright (c) 2014 The Apache Software Foundation., 
http://lucene.apache.org/solr/
+- software copyright (c) 2014 The Apache Software Foundation., 
http://mahout.apache.org/

http://git-wip-us.apache.org/repos/asf/phoenix/blob/ce6810f4/phoenix-core/pom.xml
--
diff --git a/phoenix-core/pom.xml b/phoenix-core/pom.xml
index e85ab5a..333c203 100644
--- a/phoenix-core/pom.xml
+++ b/phoenix-core/pom.xml
@@ -465,5 +465,10 @@
 joni
 ${joni.version}
 
+
+  com.clearspring.analytics
+  stream
+  2.9.5
+
   
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/ce6810f4/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
new file mode 100644
index 000..3de1509
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.*;
+import java.util.Properties;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.*;
+
+public class CountDistinctApproximateHyperLogLogIT extends 
ParallelStatsDisabledIT {
+   private String tableName;
+
+   @Before
+   public void generateTableNames() {
+   tableName = "T_" + generateUniqueName();
+   }
+
+   @Test(expected = ColumnNotFoundException.class)
+   public void testDistinctCountException() throws Exception {
+   String query = "SELECT APPROX_COUNT_DISTINCT(x) FROM " + 
tableName;
+   Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+   
+   try (Connection conn = DriverManager.getConnection(getUrl(), 
props);
+   PreparedStatement statement =

phoenix git commit: PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)

2017-08-28 Thread jamestaylor 
Repository: phoenix
Updated Branches:
  refs/heads/master 435441ea8 -> d6381afc3


PHOENIX-418 Support approximate COUNT DISTINCT (Ethan Wang)


Project: http://git-wip-us.apache.org/repos/asf/phoenix/repo
Commit: http://git-wip-us.apache.org/repos/asf/phoenix/commit/d6381afc
Tree: http://git-wip-us.apache.org/repos/asf/phoenix/tree/d6381afc
Diff: http://git-wip-us.apache.org/repos/asf/phoenix/diff/d6381afc

Branch: refs/heads/master
Commit: d6381afc3af976ccdbb874d4458ea17b1e8a1d32
Parents: 435441e
Author: James Taylor 
Authored: Mon Aug 28 15:17:37 2017 -0700
Committer: James Taylor 
Committed: Mon Aug 28 15:17:37 2017 -0700

--
 dev/release_files/NOTICE|   8 +
 phoenix-core/pom.xml|   5 +
 .../CountDistinctApproximateHyperLogLogIT.java  | 154 +++
 .../phoenix/expression/ExpressionType.java  |   4 +-
 ...stinctCountHyperLogLogAggregateFunction.java | 192 +++
 ...tinctCountHyperLogLogAggregateParseNode.java |  39 
 6 files changed, 401 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/phoenix/blob/d6381afc/dev/release_files/NOTICE
--
diff --git a/dev/release_files/NOTICE b/dev/release_files/NOTICE
index eed1edc..b7b40c0 100644
--- a/dev/release_files/NOTICE
+++ b/dev/release_files/NOTICE
@@ -277,3 +277,11 @@ jcip-annotations
 findbugs-annotations
 
   Copyright Stephen Connolly.
+
+stream-lib
+
+  Copyright 2016 AddThis.
+  This product includes software developed by AddThis.
+  This product also includes code adapted from:
+- software copyright (c) 2014 The Apache Software Foundation., 
http://lucene.apache.org/solr/
+- software copyright (c) 2014 The Apache Software Foundation., 
http://mahout.apache.org/

http://git-wip-us.apache.org/repos/asf/phoenix/blob/d6381afc/phoenix-core/pom.xml
--
diff --git a/phoenix-core/pom.xml b/phoenix-core/pom.xml
index 86c92fa..82cdacc 100644
--- a/phoenix-core/pom.xml
+++ b/phoenix-core/pom.xml
@@ -470,5 +470,10 @@
 joni
 ${joni.version}
 
+
+  com.clearspring.analytics
+  stream
+  2.9.5
+
   
 

http://git-wip-us.apache.org/repos/asf/phoenix/blob/d6381afc/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
--
diff --git 
a/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
new file mode 100644
index 000..3de1509
--- /dev/null
+++ 
b/phoenix-core/src/it/java/org/apache/phoenix/end2end/CountDistinctApproximateHyperLogLogIT.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.end2end;
+
+import org.apache.phoenix.schema.ColumnNotFoundException;
+import org.apache.phoenix.util.PropertiesUtil;
+import org.apache.phoenix.util.QueryUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.*;
+import java.util.Properties;
+
+import static org.apache.phoenix.util.TestUtil.TEST_PROPERTIES;
+import static org.junit.Assert.*;
+
+public class CountDistinctApproximateHyperLogLogIT extends 
ParallelStatsDisabledIT {
+   private String tableName;
+
+   @Before
+   public void generateTableNames() {
+   tableName = "T_" + generateUniqueName();
+   }
+
+   @Test(expected = ColumnNotFoundException.class)
+   public void testDistinctCountException() throws Exception {
+   String query = "SELECT APPROX_COUNT_DISTINCT(x) FROM " + 
tableName;
+   Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
+   
+   try (Connection conn = DriverManager.getConnection(getUrl(), 
props);
+   PreparedStatement statement =

Build failed in Jenkins: Phoenix Compile Compatibility with HBase #389

2017-08-28 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H25 (couchdbtest ubuntu xenial) in workspace 

[Phoenix_Compile_Compat_wHBase] $ /bin/bash /tmp/jenkins5810713768431602517.sh
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 386417
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 6
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 10240
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited
core id : 0
core id : 1
core id : 2
core id : 3
core id : 4
core id : 5
physical id : 0
physical id : 1
MemTotal:   98958120 kB
MemFree:26585712 kB
Filesystem  Size  Used Avail Use% Mounted on
udev 48G 0   48G   0% /dev
tmpfs   9.5G  1.1G  8.4G  12% /run
/dev/sda1   364G  150G  196G  44% /
tmpfs48G  516K   48G   1% /dev/shm
tmpfs   5.0M 0  5.0M   0% /run/lock
tmpfs48G 0   48G   0% /sys/fs/cgroup
tmpfs   9.5G 0  9.5G   0% /run/user/10025
apache-maven-2.2.1
apache-maven-3.0.4
apache-maven-3.0.5
apache-maven-3.2.1
apache-maven-3.2.5
apache-maven-3.3.3
apache-maven-3.3.9
apache-maven-3.5.0
latest
latest2
latest3


===
Verifying compile level compatibility with HBase 0.98 with Phoenix 
4.x-HBase-0.98
===

Cloning into 'hbase'...
Switched to a new branch '0.98'
Branch 0.98 set up to track remote branch 0.98 from origin.

main:
 [exec] 
~/jenkins-slave/workspace/Phoenix_Compile_Compat_wHBase/hbase/hbase-common 
~/jenkins-slave/workspace/Phoenix_Compile_Compat_wHBase/hbase/hbase-common
 [exec] 
~/jenkins-slave/workspace/Phoenix_Compile_Compat_wHBase/hbase/hbase-common

main:
[mkdir] Created dir: 

 [exec] tar: hadoop-snappy-nativelibs.tar: Cannot open: No such file or 
directory
 [exec] tar: Error is not recoverable: exiting now
 [exec] Result: 2

main:
[mkdir] Created dir: 

 [copy] Copying 20 files to 

[mkdir] Created dir: 

[mkdir] Created dir: 


main:
[mkdir] Created dir: 

 [copy] Copying 17 files to 

[mkdir] Created dir: 


main:
[mkdir] Created dir: 

 [copy] Copying 1 file to 

[mkdir] Created dir: 


HBase pom.xml:

Got HBase version as 0.98.25-SNAPSHOT
Cloning into 'phoenix'...
Switched to a new branch '4.x-HBase-0.98'
Branch 4.x-HBase-0.98 set up to track remote branch 4.x-HBase-0.98 from origin.
ANTLR Parser Generator  Version 3.5.2
Output file 

 does not exist: must build 

PhoenixSQL.g


===
Verifying compile level compatibility with HBase branch-1.3 with Phoenix master