date:20180511

[drill] branch master updated: DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp types. (#3)

2018-05-11 Thread amansinha

This is an automated email from the ASF dual-hosted git repository.

amansinha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git


The following commit(s) were added to refs/heads/master by this push:
 new c1f0adc  DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill 
Date, Time, Timestamp types. (#3)
c1f0adc is described below

commit c1f0adc9276ab8314ce9c5dff2d9c92066c71530
Author: jiang-wu 
AuthorDate: Tue May 1 14:48:26 2018 -0700

DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, 
Timestamp types. (#3)

close apache/drill#1247

* DRILL-6242 - Use java.time.Local{Date|Time|DateTime} classes to hold 
values from corresponding Drill date, time, and timestamp types.
Conflicts:

exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/ExtendedJsonOutput.java

Fix merge conflicts and check style.
---
 .../exec/TestHiveDrillNativeParquetReader.java |  20 +-
 .../drill/exec/fn/hive/TestInbuiltHiveUDFs.java|  15 +-
 .../apache/drill/exec/hive/TestHiveStorage.java|  37 ++--
 .../exec/store/jdbc/TestJdbcPluginWithDerbyIT.java |  12 +-
 .../exec/store/jdbc/TestJdbcPluginWithMySQLIT.java |  15 +-
 .../main/codegen/templates/CastDateVarChar.java|   8 +-
 .../main/codegen/templates/CastVarCharDate.java|   8 +-
 .../drill/exec/expr/fn/impl/DateTypeFunctions.java |   8 +-
 .../exec/planner/common/DrillValuesRelBase.java|  30 +--
 .../drill/exec/store/bson/BsonRecordReader.java|  25 ++-
 .../org/apache/drill/exec/util/VectorUtil.java |  10 +-
 .../exec/vector/complex/fn/BasicJsonOutput.java|  36 ++--
 .../exec/vector/complex/fn/ExtendedJsonOutput.java |  15 +-
 .../drill/exec/vector/complex/fn/JsonOutput.java   |   8 +-
 .../drill/exec/vector/complex/fn/VectorOutput.java |  34 +--
 .../java/org/apache/drill/TestFunctionsQuery.java  |  22 +-
 .../java/org/apache/drill/TestStarQueries.java |   2 +-
 .../drill/exec/fn/impl/TestCastFunctions.java  |  27 +--
 .../drill/exec/fn/impl/TestDateAddFunctions.java   |  29 +--
 .../drill/exec/fn/impl/TestDateTruncFunctions.java | 139 ++--
 .../drill/exec/fn/impl/TestNewDateFunctions.java   |  10 +-
 .../exec/fn/interp/ExpressionInterpreterTest.java  |   8 +-
 .../exec/physical/impl/TestConvertFunctions.java   |  13 +-
 .../physical/impl/TestNestedDateTimeTimestamp.java | 234 +
 .../impl/limit/TestEarlyLimit0Optimization.java|  16 +-
 .../writer/TestCorruptParquetDateCorrection.java   |  66 +++---
 .../physical/impl/writer/TestParquetWriter.java|  32 +--
 .../planner/logical/TestCaseNullableTypes.java |   5 +-
 .../drill/exec/store/avro/AvroFormatTest.java  |  75 ---
 .../exec/store/bson/TestBsonRecordReader.java  |   5 +-
 .../exec/store/parquet/TestParquetComplex.java |  52 ++---
 .../store/parquet2/TestDrillParquetReader.java |  51 ++---
 .../vector/complex/writer/TestExtendedTypes.java   |   2 +-
 .../java/org/apache/drill/test/TestBuilder.java|  19 +-
 .../test/resources/baseline_nested_datetime.json   |  13 ++
 exec/java-exec/src/test/resources/datetime.parquet | Bin 0 -> 1217 bytes
 .../src/main/codegen/data/ValueVectorTypes.tdd |   6 +-
 .../src/main/codegen/includes/vv_imports.ftl   |   8 +-
 .../codegen/templates/AbstractFieldReader.java |   2 +-
 .../main/codegen/templates/FixedValueVectors.java  |  15 +-
 .../main/codegen/templates/HolderReaderImpl.java   |  12 ++
 .../src/main/codegen/templates/NullReader.java |   2 +-
 .../src/main/codegen/templates/UnionReader.java|   2 +-
 .../drill/exec/expr/fn/impl/DateUtility.java   | 107 --
 .../drill/exec/util/JsonStringArrayList.java   |   1 +
 .../apache/drill/exec/util/JsonStringHashMap.java  |   1 +
 .../drill/exec/util/SerializationModule.java   |  75 +++
 47 files changed, 895 insertions(+), 437 deletions(-)

diff --git 
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHiveDrillNativeParquetReader.java
 
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHiveDrillNativeParquetReader.java
index 23c67b5..fd9701c 100644
--- 
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHiveDrillNativeParquetReader.java
+++ 
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/TestHiveDrillNativeParquetReader.java
@@ -17,14 +17,19 @@
  */
 package org.apache.drill.exec;
 
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+
+import java.math.BigDecimal;
+
 import org.apache.drill.PlanTestBase;
 import org.apache.drill.categories.HiveStorageTest;
 import org.apache.drill.categories.SlowTest;
 import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.exec.expr.fn.impl.DateUtility;
 import org.apache.drill.exec.hive.HiveTestBase;
 import org.apache.drill.exec.planner.physical.PlannerSettings;

[drill] 01/04: DRILL-6249: Adding more unit testing documentation.

2018-05-11 Thread amansinha

This is an automated email from the ASF dual-hosted git repository.

amansinha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git

commit dc8d0107cb6a4a31e1beaa9c98d6e51cf7486a67
Author: Timothy Farkas 
AuthorDate: Thu Mar 15 14:27:55 2018 -0700

DRILL-6249: Adding more unit testing documentation.

close apache/drill#1251
---
 README.md  |   5 +-
 .../java/org/apache/drill/test/DirTestWatcher.java |  14 +-
 .../test/java/org/apache/drill/test/TestTools.java |   7 +-
 docs/dev/BaseTestQuery.md  |   4 +
 docs/dev/ClusterTest.md|   4 +
 docs/dev/DevDocs.md|   6 +-
 docs/dev/GeneratedCode.md  |  51 +++
 docs/dev/InstantiatingComponents.md| 107 ++
 docs/dev/LicenseHeaders.md |  34 +
 docs/dev/PhysicalOpUnitTestBase.md |  31 
 docs/dev/RowSetFramework.md| 112 +++
 docs/dev/TempDirectories.md|  17 +++
 docs/dev/TestDataSets.md   | 159 +
 docs/dev/Testing.md| 134 +++--
 .../drill/exec/ops/ExchangeFragmentContext.java|   3 +
 .../drill/exec/ops/ExecutorFragmentContext.java|   4 +
 .../org/apache/drill/exec/ops/FragmentContext.java |  10 +-
 .../apache/drill/exec/ops/FragmentContextImpl.java |  28 +++-
 .../apache/drill/exec/ops/RootFragmentContext.java |   3 +
 .../org/apache/drill/exec/record/BatchSchema.java  |   4 +
 .../org/apache/drill/TestDynamicUDFSupport.java|   2 +-
 .../exec/expr/fn/FunctionInitializerTest.java  |   2 +-
 .../physical/impl/window/GenerateTestData.java |   4 +-
 .../store/dfs/TestSchemaNotFoundException.java |   8 +-
 .../org/apache/drill/test/BaseDirTestWatcher.java  |  16 ++-
 .../java/org/apache/drill/test/BaseTestQuery.java  |   4 +
 .../java/org/apache/drill/test/ExampleTest.java|  24 
 .../apache/drill/test/rowSet/RowSetBuilder.java|   7 +
 .../apache/drill/test/rowSet/RowSetUtilities.java  |   6 +
 .../drill/test/rowSet/file/JsonFileBuilder.java|  26 
 30 files changed, 794 insertions(+), 42 deletions(-)

diff --git a/README.md b/README.md
index a30cf07..f2b5b9d 100644
--- a/README.md
+++ b/README.md
@@ -20,7 +20,10 @@ Please see the [Apache Drill 
Website](http://drill.apache.org/) or the [Apache D
 
 
 ## Join the community!
-Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.  Please say hello on the Apache Drill mailing list or join our 
Google Hangouts for more information.  (More information can be found at the 
Apache Drill website).
+Apache Drill is an Apache Foundation project and is seeking all types of 
contributions.
+Please say hello on the [Apache Drill mailing 
list](http://drill.apache.org/mailinglists/)
+or join our [Google Hangouts](http://drill.apache.org/community-resources/) 
for more information.
+(More information can be found at the [Apache Drill 
website](http://drill.apache.org/)).
 
 ## Export Control
 This distribution includes cryptographic software. The country in which you 
currently reside may have restrictions on the import, possession, use, and/or 
re-export to another country, of encryption software. BEFORE using any 
encryption software, please check your country's laws, regulations and policies 
concerning the import, possession, or use, and re-export of encryption 
software, to see if this is permitted. See  for more 
information.  
diff --git a/common/src/test/java/org/apache/drill/test/DirTestWatcher.java 
b/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
index 271ac94..6bdc951 100644
--- a/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
+++ b/common/src/test/java/org/apache/drill/test/DirTestWatcher.java
@@ -31,10 +31,10 @@ import java.nio.file.Paths;
  * 
  *
  * 
- * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule} is
- * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test} annotation. When the
+ * A {@link DirTestWatcher} is added to a test by declaring it as a JUnit 
{@link org.junit.Rule}. A {@link org.junit.Rule Rule} is
+ * a piece of code that is run before and after every JUnit test marked with 
the {@link org.junit.Test Test} annotation. When the
  * {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher} 
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each 
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test Test}s and optionally delete the temp directory after 
each of your {@link org.junit.Test Test}s. The

[drill] 03/04: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

2018-05-11 Thread amansinha

This is an automated email from the ASF dual-hosted git repository.

amansinha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git

commit 11036c556f6445bcf1418b6512ba71b44a0a6031
Author: Kunal Khatua 
AuthorDate: Fri Apr 27 16:27:45 2018 -0700

DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

As a follow up to DRILL-6289, the following improvements have been done:
1. When loading the page for the first time, the WebUI enables the shutdown 
button without actually checking the state of the Drillbits.
   The ideal behaviour should be to disable the button till the state is 
verified. [Done]
   If a Drillbit is confirmed down (i.e. not in `/state` response), it is 
marked as OFFLINE and button is disabled.
2. When shutting down the current Drillbit, the WebUI no more has access to 
the cluster state.
   The ideal behaviour here should be to fetch the state from any of the 
other Drillbits to update the status. [Done]
   With the current Drillbit down, the other bits are requested for cluster 
state info and update accordingly.
3. When a new, previously unseen Drillbit comes up, the WebUI will never 
render it because the table is statically generated during the first page load.
   The idea behaviour should be to append to the table on discovery of a 
new node. [Done]
   The new Drillbit info is injected and a prompt appears to refresh the 
page to re-populate any missing info. This also works with feature (2) 
mentioned above.

The only Java code change was to have the state response carry the address 
and http-port as a tuple, instead of the user-port (which is never used).

Updates based on review

1. Added descriptive comments to functions
2. Handled possible race condition from multiple httpRequests for cluster 
state
3. Eliminated unused stateKey variable
4. Best practice of using local (let) instead of global (var) objects, and 
substituting currentRow variable instead of the jQuery search.

Additional changes based on review

1. Random selection of Drillbits to query for state when primary Drillbit 
is down (limited to 3; with a timeout of 3 sec)
2. Indicate when an Offline Drillbit is de-registered, versus just Offline 
as per Zookeeper
3. Hide shutdown column when Authentication is enabled, but the user is NOT 
an Admin.
   When the column is visible, remote bits are disabled because
4. Metrics will be shown all the time (except HTTPS), because the 
information is available and a non-admin user would anyway not have actionable 
capabilities
5. Basic clean up on unused variables

Hide shutdown buttons for HTTPS scenarios

Fixed check for 'https'

Since, `location.protocol` returns a trailing ':' as well

Handle metrics lookup on local Drillbit for HTTPS

If the webpage is accessed via IP, the certificate is tied to the IP and 
not the FQDN, which is what is used for fetching the metrics.

close apache/drill#1241
---
 .../apache/drill/exec/server/rest/DrillRoot.java   |   3 +-
 exec/java-exec/src/main/resources/rest/index.ftl   | 319 +++--
 2 files changed, 239 insertions(+), 83 deletions(-)

diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
b/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
index 50d99b0..0a48801 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
@@ -88,7 +88,7 @@ public class DrillRoot {
 Collection drillbits = getClusterInfoJSON().getDrillbits();
 Map drillStatusMap = new HashMap();
 for (DrillbitInfo drillbit : drillbits) {
-  drillStatusMap.put(drillbit.getAddress() + "-" + drillbit.getUserPort(), 
drillbit.getState());
+  drillStatusMap.put(drillbit.getAddress() + "-" + drillbit.getHttpPort(), 
drillbit.getState());
 }
 return setResponse(drillStatusMap);
   }
@@ -239,6 +239,7 @@ public class DrillRoot {
   private Response shutdown(String resp) throws Exception {
 Map shutdownInfo = new HashMap();
 new Thread(new Runnable() {
+@Override
 public void run() {
   try {
 drillbit.close();
diff --git a/exec/java-exec/src/main/resources/rest/index.ftl 
b/exec/java-exec/src/main/resources/rest/index.ftl
index 5259e44..11d1dd2 100644
--- a/exec/java-exec/src/main/resources/rest/index.ftl
+++ b/exec/java-exec/src/main/resources/rest/index.ftl
@@ -54,9 +54,11 @@
 
   
 
-  Drillbits ${model.getDrillbits()?size}
+  Drillbits ${model.getDrillbits()?size}
+  
+  
   
-
+
   
 
   #
@@ -71,9 +73,7 @@
   Version

[drill] branch master updated (8a1a7c5 -> 16659ab)

2018-05-11 Thread amansinha

This is an automated email from the ASF dual-hosted git repository.

amansinha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git.


from 8a1a7c5  DRILL-6386: Remove unused imports and star imports.
 new dc8d010  DRILL-6249: Adding more unit testing documentation.
 new 76728e5  DRILL-6321: Customize Drill's conformance. Allow support to 
APPLY keywords
 new 11036c5  DRILL-6364: Handle Cluster Info in WebUI when existing/new 
bits restart
 new 16659ab  DRILL-6401: Precision for decimal data types may be lost for 
the case when cast with literal is used

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 README.md  |   5 +-
 .../java/org/apache/drill/test/DirTestWatcher.java |  14 +-
 .../test/java/org/apache/drill/test/TestTools.java |   7 +-
 docs/dev/BaseTestQuery.md  |   4 +
 docs/dev/ClusterTest.md|   4 +
 docs/dev/DevDocs.md|   6 +-
 docs/dev/GeneratedCode.md  |  51 
 docs/dev/InstantiatingComponents.md| 107 +++
 docs/dev/LicenseHeaders.md |  34 +++
 docs/dev/PhysicalOpUnitTestBase.md |  31 ++
 docs/dev/RowSetFramework.md| 112 
 docs/dev/TempDirectories.md|  17 ++
 docs/dev/TestDataSets.md   | 159 ++
 docs/dev/Testing.md| 134 -
 .../drill/exec/ops/ExchangeFragmentContext.java|   3 +
 .../drill/exec/ops/ExecutorFragmentContext.java|   4 +
 .../org/apache/drill/exec/ops/FragmentContext.java |  10 +-
 .../apache/drill/exec/ops/FragmentContextImpl.java |  28 +-
 .../apache/drill/exec/ops/RootFragmentContext.java |   3 +
 .../drill/exec/planner/logical/DrillOptiq.java |   4 +-
 .../sql/{FixedRange.java => DrillConformance.java} |  37 ++-
 .../drill/exec/planner/sql/DrillParserConfig.java  |   4 +-
 .../drill/exec/planner/sql/SqlConverter.java   |  34 +--
 .../org/apache/drill/exec/record/BatchSchema.java  |   4 +
 .../apache/drill/exec/server/rest/DrillRoot.java   |   3 +-
 .../exec/store/parquet/ParquetFilterBuilder.java   |   4 +-
 exec/java-exec/src/main/resources/rest/index.ftl   | 319 +++--
 .../org/apache/drill/TestDynamicUDFSupport.java|   2 +-
 .../exec/expr/fn/FunctionInitializerTest.java  |   2 +-
 .../physical/impl/window/GenerateTestData.java |   4 +-
 .../sql/TestConformance.java}  |  35 +--
 .../store/dfs/TestSchemaNotFoundException.java |   8 +-
 .../org/apache/drill/test/BaseDirTestWatcher.java  |  16 +-
 .../java/org/apache/drill/test/BaseTestQuery.java  |   4 +
 .../java/org/apache/drill/test/ExampleTest.java|  24 ++
 .../apache/drill/test/rowSet/RowSetBuilder.java|   7 +
 .../apache/drill/test/rowSet/RowSetUtilities.java  |   6 +
 .../drill/test/rowSet/file/JsonFileBuilder.java|  26 ++
 .../drill/common/expression/ValueExpressions.java  |  16 +-
 39 files changed, 1105 insertions(+), 187 deletions(-)
 create mode 100644 docs/dev/BaseTestQuery.md
 create mode 100644 docs/dev/ClusterTest.md
 create mode 100644 docs/dev/GeneratedCode.md
 create mode 100644 docs/dev/InstantiatingComponents.md
 create mode 100644 docs/dev/LicenseHeaders.md
 create mode 100644 docs/dev/PhysicalOpUnitTestBase.md
 create mode 100644 docs/dev/RowSetFramework.md
 create mode 100644 docs/dev/TempDirectories.md
 create mode 100644 docs/dev/TestDataSets.md
 copy 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/{FixedRange.java 
=> DrillConformance.java} (54%)
 copy exec/java-exec/src/test/java/org/apache/drill/{TestImplicitCasting.java 
=> exec/sql/TestConformance.java} (60%)

-- 
To stop receiving notification emails like this one, please contact
amansi...@apache.org.

[drill] 04/04: DRILL-6401: Precision for decimal data types may be lost for the case when cast with literal is used

2018-05-11 Thread amansinha

This is an automated email from the ASF dual-hosted git repository.

amansinha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/drill.git

commit 16659ab30fd9fca5af39edd22aaab36409a50930
Author: Volodymyr Vysotskyi 
AuthorDate: Thu May 10 17:06:28 2018 +0300

DRILL-6401: Precision for decimal data types may be lost for the case when 
cast with literal is used

close apache/drill#1254
---
 .../apache/drill/exec/planner/logical/DrillOptiq.java|  4 +++-
 .../drill/exec/store/parquet/ParquetFilterBuilder.java   |  4 +++-
 .../apache/drill/common/expression/ValueExpressions.java | 16 ++--
 3 files changed, 16 insertions(+), 8 deletions(-)

diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
index fd791dd..2cd6068 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
@@ -550,7 +550,9 @@ public class DrillOptiq {
 literal.getType().getScale(),
 literal.getType().getPrecision()));
   }
-  return ValueExpressions.getVarDecimal((BigDecimal) 
literal.getValue());
+  return ValueExpressions.getVarDecimal((BigDecimal) 
literal.getValue(),
+  literal.getType().getPrecision(),
+  literal.getType().getScale());
 }
 double dbl = ((BigDecimal) literal.getValue()).doubleValue();
 logger.warn("Converting exact decimal into approximate decimal.\n" +
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
index 52c4723..e55425e 100644
--- 
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFilterBuilder.java
@@ -181,7 +181,9 @@ public class ParquetFilterBuilder extends 
AbstractExprVisitor

[drill] branch master updated: DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp types. (#3)

[drill] 01/04: DRILL-6249: Adding more unit testing documentation.

[drill] 03/04: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

[drill] branch master updated (8a1a7c5 -> 16659ab)

[drill] 04/04: DRILL-6401: Precision for decimal data types may be lost for the case when cast with literal is used

5 matches

Site Navigation

Mail list logo

Footer information