drill git commit: DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, Operatortypes, MockRecordBatch for testing

2018-04-16 Thread parthc 
Repository: drill
Updated Branches:
  refs/heads/master f8691f4f9 -> 1bb292072


DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, Operatortypes, 
MockRecordBatch for testing

  Added new Iterator State EMIT, added operatos LATERA_JOIN & UNNEST in 
CoreOperatorType and added LateralContract interface

  Implementation of MockRecordBatch to test operator behavior for different 
IterOutcomes. a) Creates new output container for schema change cases. b) 
Doesn't create new container for each next() call without schema change, since 
the operator in test expects the ValueVector object in it's incoming batch to 
be same unless a OK_NEW_SCHEMA case is hit. Since setup() method of operator in 
test will store the reference to value vector received in first batch

This closes #1211


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/1bb29207
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/1bb29207
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/1bb29207

Branch: refs/heads/master
Commit: 1bb292072f249bc8c4334313af8f8537c7ed1622
Parents: f8691f4
Author: Sorabh Hamirwasia 
Authored: Mon Feb 5 13:12:15 2018 -0800
Committer: Parth Chandra 
Committed: Mon Apr 16 17:10:35 2018 -0700

--
 .../exec/physical/base/LateralContract.java |  46 +
 .../apache/drill/exec/record/RecordBatch.java   |  27 ++-
 .../exec/physical/impl/MockRecordBatch.java | 185 +++
 .../org/apache/drill/test/OperatorFixture.java  |  28 +--
 .../apache/drill/exec/proto/UserBitShared.java  |  31 +++-
 .../exec/proto/beans/CoreOperatorType.java  |   6 +-
 protocol/src/main/protobuf/UserBitShared.proto  |   2 +
 7 files changed, 306 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/1bb29207/exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/LateralContract.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/LateralContract.java
 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/LateralContract.java
new file mode 100644
index 000..3d6a3c5
--- /dev/null
+++ 
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/LateralContract.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.drill.exec.physical.base;
+
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.RecordBatch.IterOutcome;
+
+/**
+ * Contract between Lateral Join and any operator on right side of it 
consuming the input
+ * from left side.
+ */
+public interface LateralContract {
+
+  /**
+   * Get reference to left side incoming of LateralJoinRecordBatch
+   * @return
+   */
+  RecordBatch getIncoming();
+
+  /**
+   * Get current record index in incoming to be processed
+   * @return
+   */
+  int getRecordIndex();
+
+  /**
+   * Get the current outcome of left incoming batch
+   */
+  IterOutcome getLeftOutcome();
+}

http://git-wip-us.apache.org/repos/asf/drill/blob/1bb29207/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java
--
diff --git 
a/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java 
b/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java
index 7fc086d..fe7f9e9 100644
--- a/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java
+++ b/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java
@@ -198,7 +198,32 @@ public interface RecordBatch extends VectorAccessible {
  * {@code OUT_OF_MEMORY} to its caller) and call {@code next()} again.
  * 
  */
-OUT_OF_MEMORY
+OUT_OF_MEMORY,
+
+/**
+ * Emit record to produce output batches.
+ * 
+ *   The call to {@link #next()},
+ *   read zero or more records with no change in schema as compared to last
+ *

drill-site git commit: doc edits

2018-04-16 Thread bridgetb 
Repository: drill-site
Updated Branches:
  refs/heads/asf-site 647e841cc -> cc35e2ff3


doc edits


Project: http://git-wip-us.apache.org/repos/asf/drill-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill-site/commit/cc35e2ff
Tree: http://git-wip-us.apache.org/repos/asf/drill-site/tree/cc35e2ff
Diff: http://git-wip-us.apache.org/repos/asf/drill-site/diff/cc35e2ff

Branch: refs/heads/asf-site
Commit: cc35e2ff3ede7dbde1df38f1c552a7e66bc6377e
Parents: 647e841
Author: Bridget Bevens 
Authored: Mon Apr 16 16:19:06 2018 -0700
Committer: Bridget Bevens 
Committed: Mon Apr 16 16:19:06 2018 -0700

--
 docs/apache-drill-1-13-0-release-notes/index.html |  2 +-
 docs/compiling-drill-from-source/index.html   |  4 ++--
 docs/distributed-mode-prerequisites/index.html|  4 ++--
 docs/embedded-mode-prerequisites/index.html   |  4 ++--
 docs/hive-storage-plugin/index.html   | 10 +-
 feed.xml  |  4 ++--
 6 files changed, 14 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill-site/blob/cc35e2ff/docs/apache-drill-1-13-0-release-notes/index.html
--
diff --git a/docs/apache-drill-1-13-0-release-notes/index.html 
b/docs/apache-drill-1-13-0-release-notes/index.html
index e5d6179..8f16043 100644
--- a/docs/apache-drill-1-13-0-release-notes/index.html
+++ b/docs/apache-drill-1-13-0-release-notes/index.html
@@ -1240,7 +1240,7 @@
 Ability to run Drill under YARN. (https://issues.apache.org/jira/browse/DRILL-1170;>DRILL-1170)
 Parquet filter pushdown support for IS [NOT] NULL, TRUE, and FALSE 
operators and implicit and explicit casts for timestamp, date, and time data 
types. (https://issues.apache.org/jira/browse/DRILL-6174;>DRILL-6174)
 Performance improvements with support for project push down, filter push 
down, and partition pruning on dynamically expanded columns when represented as 
a star in the ITEM operator. (https://issues.apache.org/jira/browse/DRILL-6118;>DRILL-6118)
-Updated Hive libraries and the Drill Hive client updated to 2.3.2 with 
support for querying Hive transactional ORC bucketed tables. (https://issues.apache.org/jira/browse/DRILL-5978;>DRILL-5978)
+The Hive client for Drill is updated to version 2.3.2. With the update, 
Drill supports queries on transactional (ACID) and non-transactional Hive 
bucketed ORC tables. The updated libraries are backward compatible with earlier 
versions of the Hive server and metastore. (https://issues.apache.org/jira/browse/DRILL-5978;>DRILL-5978)
 Ability to automatically manage memory allocations during Drill startup. 
(https://issues.apache.org/jira/browse/DRILL-5741;>DRILL-5741)
 Ability to query an empty directory and use it for queries with any JOIN 
and UNION (UNION ALL) operators. (https://issues.apache.org/jira/browse/DRILL-4185;>Drill-4185)
 Non-numeric support for JSON processing. (https://issues.apache.org/jira/browse/DRILL-5919;>Drill-5919)

http://git-wip-us.apache.org/repos/asf/drill-site/blob/cc35e2ff/docs/compiling-drill-from-source/index.html
--
diff --git a/docs/compiling-drill-from-source/index.html 
b/docs/compiling-drill-from-source/index.html
index 8accdae..cac6d4f 100644
--- a/docs/compiling-drill-from-source/index.html
+++ b/docs/compiling-drill-from-source/index.html
@@ -1218,7 +1218,7 @@
 
 
 
- 
+ Apr 16, 2018
 
 
 
@@ -1233,7 +1233,7 @@ patch review tool.
 
 
 Maven 3.0.4 or later
-Oracle JDK 7 or later
+Oracle or OpenJDK 8 
 
 
 Run the following commands to verify that you have the correct versions of

http://git-wip-us.apache.org/repos/asf/drill-site/blob/cc35e2ff/docs/distributed-mode-prerequisites/index.html
--
diff --git a/docs/distributed-mode-prerequisites/index.html 
b/docs/distributed-mode-prerequisites/index.html
index 5de9a24..7a24a79 100644
--- a/docs/distributed-mode-prerequisites/index.html
+++ b/docs/distributed-mode-prerequisites/index.html
@@ -1218,7 +1218,7 @@
 
 
 
- Mar 21, 2018
+ Apr 16, 2018
 
 
 
@@ -1232,7 +1232,7 @@ run it in a clustered environment.
 Before you install Drill on nodes in a cluster, ensure that the cluster 
meets the following prerequisites:
 
 
-(Required) Running Oracle JDK http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html;>version
 8.
+(Required) Running Oracle or OpenJDK 8
 (Required) Running a https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html#sc_RunningReplicatedZooKeeper;>ZooKeeper
 quorum
 (Recommended) Running a Hadoop cluster
 (Recommended) Using DNS

drill git commit: doc edits

2018-04-16 Thread bridgetb 
Repository: drill
Updated Branches:
  refs/heads/gh-pages 00201f619 -> d73947b7c


doc edits


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d73947b7
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d73947b7
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d73947b7

Branch: refs/heads/gh-pages
Commit: d73947b7c13db921beb0999f1f49a67714c1e071
Parents: 00201f6
Author: Bridget Bevens 
Authored: Mon Apr 16 16:12:43 2018 -0700
Committer: Bridget Bevens 
Committed: Mon Apr 16 16:12:43 2018 -0700

--
 .../develop-drill/010-compiling-drill-from-source.md | 4 ++--
 _docs/install/045-distributed-mode-prerequisites.md  | 4 ++--
 .../010-embedded-mode-prerequisites.md   | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/d73947b7/_docs/developer-information/develop-drill/010-compiling-drill-from-source.md
--
diff --git 
a/_docs/developer-information/develop-drill/010-compiling-drill-from-source.md 
b/_docs/developer-information/develop-drill/010-compiling-drill-from-source.md
index f6003ad..890317e 100644
--- 
a/_docs/developer-information/develop-drill/010-compiling-drill-from-source.md
+++ 
b/_docs/developer-information/develop-drill/010-compiling-drill-from-source.md
@@ -1,6 +1,6 @@
 ---
 title: "Compiling Drill from Source"
-date:  
+date: 2018-04-16 23:12:44 UTC
 parent: "Develop Drill"
 ---
 To develop Drill, you compile Drill from source code and then set up a project
@@ -11,7 +11,7 @@ patch review tool.
 ## Prerequisites
 
   * Maven 3.0.4 or later
-  * Oracle JDK 7 or later
+  * Oracle or OpenJDK 8 
 
 Run the following commands to verify that you have the correct versions of
 Maven and JDK installed:

http://git-wip-us.apache.org/repos/asf/drill/blob/d73947b7/_docs/install/045-distributed-mode-prerequisites.md
--
diff --git a/_docs/install/045-distributed-mode-prerequisites.md 
b/_docs/install/045-distributed-mode-prerequisites.md
index 91d597a..5e3bf48 100644
--- a/_docs/install/045-distributed-mode-prerequisites.md
+++ b/_docs/install/045-distributed-mode-prerequisites.md
@@ -1,6 +1,6 @@
 ---
 title: "Distributed Mode Prerequisites"
-date: 2018-03-21 01:38:36 UTC
+date: 2018-04-16 23:12:44 UTC
 parent: "Installing Drill in Distributed Mode"
 ---
 You can install Apache Drill on one or more nodes to
@@ -10,7 +10,7 @@ run it in a clustered environment.
 
 Before you install Drill on nodes in a cluster, ensure that the cluster meets 
the following prerequisites:
 
-  * (Required) Running Oracle JDK [version 
8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html).
   
+  * (Required) Running Oracle or OpenJDK 8
   * (Required) Running a [ZooKeeper 
quorum](https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html#sc_RunningReplicatedZooKeeper)
  
   * (Recommended) Running a Hadoop cluster   
   * (Recommended) Using DNS 

http://git-wip-us.apache.org/repos/asf/drill/blob/d73947b7/_docs/install/installing-drill-in-embedded-mode/010-embedded-mode-prerequisites.md
--
diff --git 
a/_docs/install/installing-drill-in-embedded-mode/010-embedded-mode-prerequisites.md
 
b/_docs/install/installing-drill-in-embedded-mode/010-embedded-mode-prerequisites.md
index 4c1cdc8..777dba0 100644
--- 
a/_docs/install/installing-drill-in-embedded-mode/010-embedded-mode-prerequisites.md
+++ 
b/_docs/install/installing-drill-in-embedded-mode/010-embedded-mode-prerequisites.md
@@ -1,13 +1,13 @@
 ---
 title: "Embedded Mode Prerequisites"
-date: 2018-03-21 01:38:36 UTC
+date: 2018-04-16 23:12:45 UTC
 parent: "Installing Drill in Embedded Mode"
 ---
 To use Drill on a single node, install Drill in embedded mode. Installing 
Drill in embedded mode installs Drill locally on your machine. Embedded mode is 
a quick way to install and try Drill without having to perform any 
configuration tasks. A ZooKeeper installation is not required. Installing Drill 
in embedded mode configures the local Drillbit service to start automatically 
when you launch the Drill shell. You can install Drill in embedded mode on a 
machine running Linux, Mac OS X, or Windows.
 
 Before you install Drill, ensure that the machine meets the following 
prerequisites:
 
-* Linux, Mac OS X, and Windows: Oracle JDK [version 
8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html).
  
+* Linux, Mac OS X, and Windows: Oracle or OpenJDK 8  
 * Windows only:  
   * A JAVA_HOME environment variable that points to the JDK

drill git commit: dox edits

2018-04-16 Thread bridgetb 
Repository: drill
Updated Branches:
  refs/heads/gh-pages f511ae086 -> 00201f619


dox edits


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/00201f61
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/00201f61
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/00201f61

Branch: refs/heads/gh-pages
Commit: 00201f6195f4cd02d092cc82d19161c821adb04f
Parents: f511ae0
Author: Bridget Bevens 
Authored: Mon Apr 16 16:02:12 2018 -0700
Committer: Bridget Bevens 
Committed: Mon Apr 16 16:02:12 2018 -0700

--
 .../plugins/070-hive-storage-plugin.md| 10 +-
 _docs/rn/001-1.13.0-rn.md |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/00201f61/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md
--
diff --git a/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md 
b/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md
index 6fc5fac..11c01ff 100644
--- a/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md
+++ b/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md
@@ -1,16 +1,16 @@
 ---
 title: "Hive Storage Plugin"
-date: 2016-01-08
+date: 2018-04-16 23:02:13 UTC
 parent: "Connect a Data Source"
 ---
-Drill 1.1 and later supports Hive 1.0. To access Hive tables
-using custom SerDes or InputFormat/OutputFormat, all nodes running Drillbits
-must have the SerDes or InputFormat/OutputFormat `JAR` files in the 
+Prior to Drill 1.13, Drill supported Hive 1.0. Drill 1.13 and later includes 
version 2.3.2 of the Hive client, which adds support for queries on 
transactional (ACID) and non-transactional Hive bucketed ORC tables. The 
updated Hive libraries are backward compatible with earlier versions of the 
Hive server and metastore.   
+
+To access Hive tables using custom SerDes or InputFormat/OutputFormat, all 
nodes running Drillbits must have the SerDes or InputFormat/OutputFormat `JAR` 
files in the 
 `/jars/3rdparty` folder.
 
 You can run Hive queries in the following ways by configuring the Hive storage 
plugin as described in this document:
 
-* [Connect Drill to the Hive remote 
metastore]({{site.baseurl}}/docs/hive-storage-plugin/#connect-drill-to-the-hive-remote-metastore-directly)
  
+* [Connect Drill to the Hive remote 
metastore]({{site.baseurl}}/docs/hive-storage-plugin/#connect-drill-to-the-hive-remote-metastore)
  
 * [Connect to the Hive embedded 
metastore]({{site.baseurl}}/docs/hive-storage-plugin/#connect-to-the-hive-embedded-metastore)
  
 
 You update the Hive storage plugin by selecting the **Storage tab** on the 
[Drill Web Console]({{ site.baseurl 
}}/docs/plugin-configuration-basics/#using-the-drill-web-console). From the 
list of disabled storage plugins in the Drill Web Console, click **Update** 
next to `hive`.  The default Hive storage plugin configuration appears as 
follows:

http://git-wip-us.apache.org/repos/asf/drill/blob/00201f61/_docs/rn/001-1.13.0-rn.md
--
diff --git a/_docs/rn/001-1.13.0-rn.md b/_docs/rn/001-1.13.0-rn.md
index 71638f7..a2e1ab2 100644
--- a/_docs/rn/001-1.13.0-rn.md
+++ b/_docs/rn/001-1.13.0-rn.md
@@ -19,7 +19,7 @@ This release of Drill provides the following new features and 
improvements:
 - Ability to run [Drill under YARN]({{site.baseurl}}/docs/drill-on-yarn/). 
([DRILL-1170](https://issues.apache.org/jira/browse/DRILL-1170))   
 - Parquet filter pushdown support for IS [NOT] NULL, TRUE, and FALSE operators 
and implicit and explicit casts for timestamp, date, and time data types. 
([DRILL-6174](https://issues.apache.org/jira/browse/DRILL-6174))  
 - Performance improvements with support for project push down, filter push 
down, and partition pruning on dynamically expanded columns when represented as 
a star in the ITEM operator. 
([DRILL-6118](https://issues.apache.org/jira/browse/DRILL-6118))  
-- Updated Hive libraries and the Drill Hive client updated to 2.3.2 with 
support for querying Hive transactional ORC bucketed tables. 
([DRILL-5978](https://issues.apache.org/jira/browse/DRILL-5978))
+- The Hive client for Drill is updated to version 2.3.2. With the update, 
Drill supports queries on transactional (ACID) and non-transactional Hive 
bucketed ORC tables. The updated libraries are backward compatible with earlier 
versions of the Hive server and metastore. 
([DRILL-5978](https://issues.apache.org/jira/browse/DRILL-5978))
 - Ability to automatically manage memory allocations during Drill startup. 
([DRILL-5741](https://issues.apache.org/jira/browse/DRILL-5741))  
 - Ability to query an empty