[hive] branch master updated: HIVE-21055 : REPL LOAD command executing copy in serial mode even if parallel execution is enabled using WITH clause. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)

2019-02-12 Thread mahesh 
This is an automated email from the ASF dual-hosted git repository.

mahesh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new b0d01eb  HIVE-21055 : REPL LOAD command executing copy in serial mode 
even if parallel execution is enabled using WITH clause. (Mahesh Kumar Behera, 
reviewed by Sankar Hariappan)
b0d01eb is described below

commit b0d01eb68a604bb6efb68455193803f0651dc265
Author: Mahesh Kumar Behera 
AuthorDate: Wed Feb 13 12:55:38 2019 +0530

HIVE-21055 : REPL LOAD command executing copy in serial mode even if 
parallel execution is enabled using WITH clause. (Mahesh Kumar Behera, reviewed 
by Sankar Hariappan)
---
 ql/src/java/org/apache/hadoop/hive/ql/Driver.java   | 2 +-
 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java| 5 +
 ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java| 4 
 .../hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java  | 6 +-
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
b/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
index 95a4653..cac14a6 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
@@ -2696,7 +2696,7 @@ public class Driver implements IDriver {
 
 cxt.launching(tskRun);
 // Launch Task
-if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.EXECPARALLEL) && 
tsk.canExecuteInParallel()) {
+if (HiveConf.getBoolVar(tsk.getConf(), HiveConf.ConfVars.EXECPARALLEL) && 
tsk.canExecuteInParallel()) {
   // Launch it in the parallel mode, as a separate thread only for MR tasks
   if (LOG.isInfoEnabled()){
 LOG.info("Starting task [" + tsk + "] in parallel");
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
index 4f2a116..1c2522d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
@@ -870,4 +870,9 @@ public class MoveTask extends Task implements 
Serializable {
   public String getName() {
 return "MOVE";
   }
+
+  @Override
+  public boolean canExecuteInParallel() {
+return false;
+  }
 }
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
index b39771f..3308797 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
@@ -435,6 +435,10 @@ public abstract class Task 
implements Serializable, Node
 this.conf = conf;
   }
 
+  public HiveConf getConf() {
+return this.conf;
+  }
+
   public void setWork(T work) {
 this.work = work;
   }
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
index 5302191..3e0c969 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
@@ -382,11 +382,7 @@ public class IncrementalLoadTasksBuilder {
 }
 
 // Link import tasks to the barrier task which will in-turn linked with 
repl state update tasks
-for (Task t : importTasks){
-  t.addDependentTask(barrierTask);
-  log.debug("Added {}:{} as a precursor of barrier task {}:{}",
-  t.getClass(), t.getId(), barrierTask.getClass(), 
barrierTask.getId());
-}
+DAGTraversal.traverse(importTasks, new AddDependencyToLeaves(barrierTask));
 
 // At least one task would have been added to update the repl state
 return tasks;

[hive] branch master updated: HIVE-21250 : NPE in HiveProtoLoggingHook for eventPerFile mode. (Harish JP, reviewd by Anishek Agarwal)

2019-02-12 Thread anishek 
This is an automated email from the ASF dual-hosted git repository.

anishek pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 5db4c77  HIVE-21250 : NPE in HiveProtoLoggingHook for eventPerFile 
mode. (Harish JP, reviewd by Anishek Agarwal)
5db4c77 is described below

commit 5db4c77699ff2adeb30ef2d8ea038cfbd9035d99
Author: Anishek Agarwal 
AuthorDate: Wed Feb 13 11:49:14 2019 +0530

HIVE-21250 : NPE in HiveProtoLoggingHook for eventPerFile mode. (Harish JP, 
reviewd by Anishek Agarwal)
---
 ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java 
b/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
index 3d51c07..ec00ab6 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
@@ -305,10 +305,10 @@ public class HiveProtoLoggingHook implements 
ExecuteWithHookContext {
   for (int retryCount = 0; retryCount <= MAX_RETRIES; ++retryCount) {
 try {
   if (eventPerFile) {
-LOG.debug("Event per file enabled. Closing proto event file: {}", 
writer.getPath());
 if (!maybeRolloverWriterForDay()) {
   writer = logger.getWriter(logFileName + "_" + ++logFileCount);
 }
+LOG.debug("Event per file enabled. New proto event file: {}", 
writer.getPath());
 writer.writeProto(event);
 IOUtils.closeQuietly(writer);
 writer = null;

[hive] branch master updated: HIVE-21238: Fix Surrogate Key issue (Miklos Gergely reviewed by Vineet Garg)

2019-02-12 Thread vgarg 
This is an automated email from the ASF dual-hosted git repository.

vgarg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 793f192  HIVE-21238: Fix Surrogate Key issue (Miklos Gergely reviewed 
by Vineet Garg)
793f192 is described below

commit 793f192de238874d32d4c2f5137e97ebf048cc70
Author: Miklos Gergely 
AuthorDate: Tue Feb 12 11:34:08 2019 -0800

HIVE-21238: Fix Surrogate Key issue (Miklos Gergely reviewed by Vineet Garg)
---
 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java | 4 
 1 file changed, 4 insertions(+)

diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index 0d0196e..f2ba1ad 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -7957,6 +7957,10 @@ public class SemanticAnalyzer extends 
BaseSemanticAnalyzer {
 }
   }
 }
+
+for (Operator parent : (List>)input.getParentOperators()) {
+  setWriteIdForSurrogateKeys(ltd, parent);
+}
   }
 
   private WriteEntity generateTableWriteEntity(String dest, Table dest_tab,

[hive] branch branch-3.1 updated: HIVE-21236: SharedWorkOptimizer should check table properties (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

2019-02-12 Thread jcamacho 
This is an automated email from the ASF dual-hosted git repository.

jcamacho pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 008eda8  HIVE-21236: SharedWorkOptimizer should check table properties 
(Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
008eda8 is described below

commit 008eda8c50cd8f8f092b676a2c756ac0162dff81
Author: Jesus Camacho Rodriguez 
AuthorDate: Mon Feb 11 16:21:48 2019 -0800

HIVE-21236: SharedWorkOptimizer should check table properties (Jesus 
Camacho Rodriguez, reviewed by Ashutosh Chauhan)
---
 .../org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java  | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
index f734f79..6198316 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
@@ -30,6 +30,7 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.Objects;
 import java.util.Set;
 import java.util.TreeMap;
 
@@ -675,6 +676,10 @@ public class SharedWorkOptimizer extends Transform {
 if (tsOp1.getConf().getRowLimit() != tsOp2.getConf().getRowLimit()) {
   return false;
 }
+// If table properties do not match, we currently do not merge
+if (!Objects.equals(tsOp1.getConf().getOpProps(), 
tsOp2.getConf().getOpProps())) {
+  return false;
+}
 // If partitions do not match, we currently do not merge
 PrunedPartitionList prevTsOpPPList = pctx.getPrunedPartitions(tsOp1);
 PrunedPartitionList tsOpPPList = pctx.getPrunedPartitions(tsOp2);
@@ -1087,7 +1092,8 @@ public class SharedWorkOptimizer extends Transform {
   && StringUtils.equals(op1Conf.getFilterExprString(), 
op2Conf.getFilterExprString())
   && pctx.getPrunedPartitions(tsOp1).getPartitions().equals(
   pctx.getPrunedPartitions(tsOp2).getPartitions())
-  && op1Conf.getRowLimit() == op2Conf.getRowLimit()) {
+  && op1Conf.getRowLimit() == op2Conf.getRowLimit()
+  && Objects.equals(op1Conf.getOpProps(), op2Conf.getOpProps())) {
 return true;
   } else {
 return false;

[hive] branch branch-3 updated: HIVE-21236: SharedWorkOptimizer should check table properties (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

2019-02-12 Thread jcamacho 
This is an automated email from the ASF dual-hosted git repository.

jcamacho pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/branch-3 by this push:
 new c791b90  HIVE-21236: SharedWorkOptimizer should check table properties 
(Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
c791b90 is described below

commit c791b901e1f579e167cda4581a6874c10ca1039e
Author: Jesus Camacho Rodriguez 
AuthorDate: Mon Feb 11 16:21:48 2019 -0800

HIVE-21236: SharedWorkOptimizer should check table properties (Jesus 
Camacho Rodriguez, reviewed by Ashutosh Chauhan)
---
 .../org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java  | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
index 07c5d38..247f9b0 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
@@ -30,6 +30,7 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.Objects;
 import java.util.Set;
 import java.util.TreeMap;
 
@@ -738,6 +739,10 @@ public class SharedWorkOptimizer extends Transform {
 if (tsOp1.getConf().getRowLimit() != tsOp2.getConf().getRowLimit()) {
   return false;
 }
+// If table properties do not match, we currently do not merge
+if (!Objects.equals(tsOp1.getConf().getOpProps(), 
tsOp2.getConf().getOpProps())) {
+  return false;
+}
 // If partitions do not match, we currently do not merge
 PrunedPartitionList prevTsOpPPList = pctx.getPrunedPartitions(tsOp1);
 PrunedPartitionList tsOpPPList = pctx.getPrunedPartitions(tsOp2);
@@ -1149,7 +1154,8 @@ public class SharedWorkOptimizer extends Transform {
   && StringUtils.equals(op1Conf.getFilterExprString(), 
op2Conf.getFilterExprString())
   && pctx.getPrunedPartitions(tsOp1).getPartitions().equals(
   pctx.getPrunedPartitions(tsOp2).getPartitions())
-  && op1Conf.getRowLimit() == op2Conf.getRowLimit()) {
+  && op1Conf.getRowLimit() == op2Conf.getRowLimit()
+  && Objects.equals(op1Conf.getOpProps(), op2Conf.getOpProps())) {
 return true;
   } else {
 return false;

[hive] branch master updated: HIVE-21236: SharedWorkOptimizer should check table properties (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

2019-02-12 Thread jcamacho 
This is an automated email from the ASF dual-hosted git repository.

jcamacho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 598cb81  HIVE-21236: SharedWorkOptimizer should check table properties 
(Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
598cb81 is described below

commit 598cb819a4ead4d2400ce14e79651275a15cebe5
Author: Jesus Camacho Rodriguez 
AuthorDate: Mon Feb 11 16:21:48 2019 -0800

HIVE-21236: SharedWorkOptimizer should check table properties (Jesus 
Camacho Rodriguez, reviewed by Ashutosh Chauhan)
---
 .../org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java  | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
index 52343f4..c953e03 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
@@ -31,6 +31,7 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.Objects;
 import java.util.Set;
 import java.util.TreeMap;
 
@@ -801,6 +802,10 @@ public class SharedWorkOptimizer extends Transform {
 if (tsOp1.getConf().getRowLimit() != tsOp2.getConf().getRowLimit()) {
   return false;
 }
+// If table properties do not match, we currently do not merge
+if (!Objects.equals(tsOp1.getConf().getOpProps(), 
tsOp2.getConf().getOpProps())) {
+  return false;
+}
 // If partitions do not match, we currently do not merge
 PrunedPartitionList prevTsOpPPList = pctx.getPrunedPartitions(tsOp1);
 PrunedPartitionList tsOpPPList = pctx.getPrunedPartitions(tsOp2);
@@ -1338,7 +1343,8 @@ public class SharedWorkOptimizer extends Transform {
   && StringUtils.equals(op1Conf.getFilterExprString(), 
op2Conf.getFilterExprString())
   && pctx.getPrunedPartitions(tsOp1).getPartitions().equals(
   pctx.getPrunedPartitions(tsOp2).getPartitions())
-  && op1Conf.getRowLimit() == op2Conf.getRowLimit()) {
+  && op1Conf.getRowLimit() == op2Conf.getRowLimit()
+  && Objects.equals(op1Conf.getOpProps(), op2Conf.getOpProps())) {
 return true;
   } else {
 return false;

[hive] branch master updated: HIVE-21212: LLAP: shuffle port config uses internal configuration (Prasanth Jayachandran reviewed by Gopal V)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new e540cf8  HIVE-21212: LLAP: shuffle port config uses internal 
configuration (Prasanth Jayachandran reviewed by Gopal V)
e540cf8 is described below

commit e540cf8238680453fc04e861dbc332e7f56b904a
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:37:26 2019 -0800

HIVE-21212: LLAP: shuffle port config uses internal configuration (Prasanth 
Jayachandran reviewed by Gopal V)
---
 .../src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java   | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
index 2279f8d..41eca8f 100644
--- 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
+++ 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
@@ -534,8 +534,7 @@ public class LlapDaemon extends CompositeService implements 
ContainerRunner, Lla
   new String[0] : StringUtils.getTrimmedStrings(localDirList);
   int rpcPort = HiveConf.getIntVar(daemonConf, 
ConfVars.LLAP_DAEMON_RPC_PORT);
   int mngPort = HiveConf.getIntVar(daemonConf, 
ConfVars.LLAP_MANAGEMENT_RPC_PORT);
-  int shufflePort = daemonConf
-  .getInt(ShuffleHandler.SHUFFLE_PORT_CONFIG_KEY, 
ShuffleHandler.DEFAULT_SHUFFLE_PORT);
+  int shufflePort = HiveConf.getIntVar(daemonConf, 
ConfVars.LLAP_DAEMON_YARN_SHUFFLE_PORT);
   int webPort = HiveConf.getIntVar(daemonConf, 
ConfVars.LLAP_DAEMON_WEB_PORT);
 
   LlapDaemonInfo.initialize(appName, daemonConf);

[hive] branch master updated: HIVE-21071: Improve getInputSummary (BELUGA BEHR via Peter Vary)

2019-02-12 Thread pvary 
This is an automated email from the ASF dual-hosted git repository.

pvary pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 7700359  HIVE-21071: Improve getInputSummary (BELUGA BEHR via Peter 
Vary)
7700359 is described below

commit 77003590688512f5a62349868b62e4ed66075184
Author: BELUGA BEHR 
AuthorDate: Tue Feb 12 09:35:34 2019 +0100

HIVE-21071: Improve getInputSummary (BELUGA BEHR via Peter Vary)
---
 .../org/apache/hadoop/hive/ql/exec/Utilities.java  | 145 ---
 .../hadoop/hive/ql/exec/TestGetInputSummary.java   | 443 +
 .../apache/hadoop/hive/ql/exec/TestUtilities.java  | 212 --
 3 files changed, 524 insertions(+), 276 deletions(-)

diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
index b84b052..61e3430 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
@@ -18,12 +18,6 @@
 
 package org.apache.hadoop.hive.ql.exec;
 
-import com.esotericsoftware.kryo.Kryo;
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
 import java.beans.DefaultPersistenceDelegate;
 import java.beans.Encoder;
 import java.beans.Expression;
@@ -53,6 +47,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Calendar;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.Enumeration;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -65,17 +60,18 @@ import java.util.Properties;
 import java.util.Random;
 import java.util.Set;
 import java.util.UUID;
-import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Callable;
 import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import java.util.zip.Deflater;
 import java.util.zip.DeflaterOutputStream;
 import java.util.zip.InflaterInputStream;
 
-import com.google.common.util.concurrent.ThreadFactoryBuilder;
-
 import org.apache.commons.codec.binary.Base64;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
@@ -138,7 +134,6 @@ import org.apache.hadoop.hive.ql.io.OneNullRowInputFormat;
 import org.apache.hadoop.hive.ql.io.RCFile;
 import org.apache.hadoop.hive.ql.io.ReworkMapredInputFormat;
 import org.apache.hadoop.hive.ql.io.SelfDescribingInputFormatInterface;
-import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedDelta;
 import org.apache.hadoop.hive.ql.io.merge.MergeFileMapper;
 import org.apache.hadoop.hive.ql.io.merge.MergeFileWork;
 import org.apache.hadoop.hive.ql.io.rcfile.truncate.ColumnTruncateMapper;
@@ -211,9 +206,12 @@ import org.apache.hive.common.util.ReflectionUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
+import com.esotericsoftware.kryo.Kryo;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
 
 
 /**
@@ -2435,36 +2433,32 @@ public final class Utilities {
 PerfLogger perfLogger = SessionState.getPerfLogger();
 perfLogger.PerfLogBegin(CLASS_NAME, PerfLogger.INPUT_SUMMARY);
 
-long[] summary = {0, 0, 0};
-
+final long[] summary = {0L, 0L, 0L};
 final Set pathNeedProcess = new HashSet<>();
 
 // Since multiple threads could call this method concurrently, locking
 // this method will avoid number of threads out of control.
 synchronized (INPUT_SUMMARY_LOCK) {
   // For each input path, calculate the total size.
-  for (Path path : work.getPathToAliases().keySet()) {
-Path p = path;
-
-if (filter != null && !filter.accept(p)) {
+  for (final Path path : work.getPathToAliases().keySet()) {
+if (path == null) {
+  continue;
+}
+if (filter != null && !filter.accept(path)) {
   continue;
 }
 
 ContentSummary cs = ctx.getCS(path);
-if (cs == null) {
-  if (path == null) {
-continue;
-  }
-  pathNeedProcess.add(path);
-} else {
+if (cs != null) {
   summary[0] += cs.getLength();
   summary[1] += cs.getFileCount();
   summary[2] += cs.getDirectoryCount();
+} else {
+  pathNeedProcess.add(path);
 }
   }
 
   //

[hive] branch master updated: HIVE-21103: PartitionManagementTask should not modify DN configs to avoid closing persistence manager (Prasanth Jayachandran reviewed by Sankar Hariappan)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 7e89da8  HIVE-21103: PartitionManagementTask should not modify DN 
configs to avoid closing persistence manager (Prasanth Jayachandran reviewed by 
Sankar Hariappan)
7e89da8 is described below

commit 7e89da8bc2730873147e47fb80a2f9e7a1e5e938
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:30:23 2019 -0800

HIVE-21103: PartitionManagementTask should not modify DN configs to avoid 
closing persistence manager (Prasanth Jayachandran reviewed by Sankar Hariappan)
---
 .../java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java| 5 +++--
 .../org/apache/hadoop/hive/metastore/PartitionManagementTask.java| 3 ---
 2 files changed, 3 insertions(+), 5 deletions(-)

diff --git 
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 
b/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
index 313f87b..7c1e30a 100644
--- 
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
+++ 
b/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
@@ -731,9 +731,10 @@ public class MetastoreConf {
   "metastore.partition.management.table.types", 
"MANAGED_TABLE,EXTERNAL_TABLE",
   "Comma separated list of table types to use for partition management"),
 
PARTITION_MANAGEMENT_TASK_THREAD_POOL_SIZE("metastore.partition.management.task.thread.pool.size",
-  "metastore.partition.management.task.thread.pool.size", 5,
+  "metastore.partition.management.task.thread.pool.size", 3,
   "Partition management uses thread pool on to which tasks are submitted 
for discovering and retaining the\n" +
-  "partitions. This determines the size of the thread pool."),
+  "partitions. This determines the size of the thread pool. Note: 
Increasing the thread pool size will cause\n" +
+  "threadPoolSize * maxConnectionPoolSize connections to backend db"),
 
PARTITION_MANAGEMENT_CATALOG_NAME("metastore.partition.management.catalog.name",
   "metastore.partition.management.catalog.name", "hive",
   "Automatic partition management will look for tables under the specified 
catalog name"),
diff --git 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
index 901bf80..59001b5 100644
--- 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
+++ 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
@@ -190,9 +190,6 @@ public class PartitionManagementTask implements 
MetastoreTaskThread {
 // when invalid path is encountered as these are background threads. We 
just want to skip and move on. Users will
 // have to fix the invalid paths via external means.
 conf.set(MetastoreConf.ConfVars.MSCK_PATH_VALIDATION.getVarname(), "skip");
-// since msck runs in thread pool and each of them create their own 
metastore client, we don't want explosion of
-// connections to metastore for embedded mode. Also we don't need too many 
db connections anyway.
-
conf.setInt(MetastoreConf.ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS.getVarname(),
 2);
   }
 
   private static class MsckThread implements Runnable {

[hive] branch master updated: HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth Jayachandran reviewed by Sergey Shelukhin)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 567830e  HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth 
Jayachandran reviewed by Sergey Shelukhin)
567830e is described below

commit 567830e9ca7d28e92cc3480602394f47a79ba308
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:28:09 2019 -0800

HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth Jayachandran 
reviewed by Sergey Shelukhin)
---
 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java | 4 
 .../hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java | 8 ++--
 .../apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java  | 6 +-
 .../hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java | 2 +-
 .../hive/llap/tezplugins/endpoint/LlapPluginServerImpl.java   | 7 +--
 5 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index 2156ff1..4a86b0a 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -4165,6 +4165,8 @@ public class HiveConf extends Configuration {
 LLAP_DAEMON_RPC_NUM_HANDLERS("hive.llap.daemon.rpc.num.handlers", 5,
   "Number of RPC handlers for LLAP daemon.", 
"llap.daemon.rpc.num.handlers"),
 
+LLAP_PLUGIN_RPC_PORT("hive.llap.plugin.rpc.port", 0,
+  "Port to use for LLAP plugin rpc server"),
 LLAP_PLUGIN_RPC_NUM_HANDLERS("hive.llap.plugin.rpc.num.handlers", 1,
   "Number of RPC handlers for AM LLAP plugin endpoint."),
 LLAP_DAEMON_WORK_DIRS("hive.llap.daemon.work.dirs", "",
@@ -4338,6 +4340,8 @@ public class HiveConf extends Configuration {
   "Sleep duration (in milliseconds) to wait before retrying on error when 
obtaining a\n" +
   "connection to LLAP daemon from Tez AM.",
   "llap.task.communicator.connection.sleep-between-retries-millis"),
+LLAP_TASK_UMBILICAL_SERVER_PORT("hive.llap.daemon.umbilical.port", 0,
+  "LLAP task umbilical server RPC port"),
 LLAP_DAEMON_WEB_PORT("hive.llap.daemon.web.port", 15002, "LLAP daemon web 
UI port.",
   "llap.daemon.service.port"),
 LLAP_DAEMON_WEB_SSL("hive.llap.daemon.web.ssl", false,
diff --git 
a/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
 
b/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
index 89cb6fb..a16c0af 100644
--- 
a/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
+++ 
b/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
@@ -26,6 +26,7 @@ import java.util.concurrent.atomic.AtomicLong;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.hive.conf.HiveConf;
 import 
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.QueryIdentifierProto;
 import org.apache.hadoop.hive.llap.protocol.LlapTaskUmbilicalProtocol;
 import org.apache.hadoop.ipc.RPC;
@@ -53,11 +54,14 @@ public class LlapTaskUmbilicalServer {
 
   public LlapTaskUmbilicalServer(Configuration conf, LlapTaskUmbilicalProtocol 
umbilical, int numHandlers) throws IOException {
 jobTokenSecretManager = new JobTokenSecretManager();
-
+int umbilicalPort = HiveConf.getIntVar(conf, 
HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT);
+if (umbilicalPort <= 0) {
+  umbilicalPort = 0;
+}
 server = new RPC.Builder(conf)
 .setProtocol(LlapTaskUmbilicalProtocol.class)
 .setBindAddress("0.0.0.0")
-.setPort(0)
+.setPort(umbilicalPort)
 .setInstance(umbilical)
 .setNumHandlers(numHandlers)
 .setSecretManager(jobTokenSecretManager).build();
diff --git 
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 
b/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
index 0120bb6..dc10f22 100644
--- 
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
+++ 
b/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
@@ -255,10 +255,14 @@ public class LlapTaskCommunicator extends 
TezTaskCommunicatorImpl {
 
   int numHandlers =
   HiveConf.getIntVar(conf, 
ConfVars.LLAP_TASK_COMMUNICATOR_LISTENER_THREAD_COUNT);
+  int umbilicalPort = HiveConf.getIntVar(conf, 
ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT);
+  if (umbilicalPort <= 0) {
+umbilicalPort = 0;
+  }
   server = new RPC.Builder(conf)
   .setProtocol(LlapTaskUmbilicalProtocol.class)
   .setBindAddress("0.0.0.0")
-

[hive] branch branch-3 updated: HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth Jayachandran reviewed by Sergey Shelukhin)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/branch-3 by this push:
 new d9b3833  HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth 
Jayachandran reviewed by Sergey Shelukhin)
d9b3833 is described below

commit d9b3833e0c7b0ee008d1dd39bacc8f758170156f
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:28:09 2019 -0800

HIVE-20841: LLAP: Make dynamic ports configurable (Prasanth Jayachandran 
reviewed by Sergey Shelukhin)
---
 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java | 4 
 .../hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java | 8 ++--
 .../apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java  | 6 +-
 .../hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java | 2 +-
 .../hive/llap/tezplugins/endpoint/LlapPluginServerImpl.java   | 7 +--
 5 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index a04ef38..3bb482f 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -4010,6 +4010,8 @@ public class HiveConf extends Configuration {
 LLAP_DAEMON_RPC_NUM_HANDLERS("hive.llap.daemon.rpc.num.handlers", 5,
   "Number of RPC handlers for LLAP daemon.", 
"llap.daemon.rpc.num.handlers"),
 
+LLAP_PLUGIN_RPC_PORT("hive.llap.plugin.rpc.port", 0,
+  "Port to use for LLAP plugin rpc server"),
 LLAP_PLUGIN_RPC_NUM_HANDLERS("hive.llap.plugin.rpc.num.handlers", 1,
   "Number of RPC handlers for AM LLAP plugin endpoint."),
 LLAP_DAEMON_WORK_DIRS("hive.llap.daemon.work.dirs", "",
@@ -4182,6 +4184,8 @@ public class HiveConf extends Configuration {
   "Sleep duration (in milliseconds) to wait before retrying on error when 
obtaining a\n" +
   "connection to LLAP daemon from Tez AM.",
   "llap.task.communicator.connection.sleep-between-retries-millis"),
+LLAP_TASK_UMBILICAL_SERVER_PORT("hive.llap.daemon.umbilical.port", 0,
+  "LLAP task umbilical server RPC port"),
 LLAP_DAEMON_WEB_PORT("hive.llap.daemon.web.port", 15002, "LLAP daemon web 
UI port.",
   "llap.daemon.service.port"),
 LLAP_DAEMON_WEB_SSL("hive.llap.daemon.web.ssl", false,
diff --git 
a/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
 
b/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
index 89cb6fb..a16c0af 100644
--- 
a/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
+++ 
b/llap-client/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/LlapTaskUmbilicalServer.java
@@ -26,6 +26,7 @@ import java.util.concurrent.atomic.AtomicLong;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.hive.conf.HiveConf;
 import 
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos.QueryIdentifierProto;
 import org.apache.hadoop.hive.llap.protocol.LlapTaskUmbilicalProtocol;
 import org.apache.hadoop.ipc.RPC;
@@ -53,11 +54,14 @@ public class LlapTaskUmbilicalServer {
 
   public LlapTaskUmbilicalServer(Configuration conf, LlapTaskUmbilicalProtocol 
umbilical, int numHandlers) throws IOException {
 jobTokenSecretManager = new JobTokenSecretManager();
-
+int umbilicalPort = HiveConf.getIntVar(conf, 
HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT);
+if (umbilicalPort <= 0) {
+  umbilicalPort = 0;
+}
 server = new RPC.Builder(conf)
 .setProtocol(LlapTaskUmbilicalProtocol.class)
 .setBindAddress("0.0.0.0")
-.setPort(0)
+.setPort(umbilicalPort)
 .setInstance(umbilical)
 .setNumHandlers(numHandlers)
 .setSecretManager(jobTokenSecretManager).build();
diff --git 
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 
b/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
index 5d4ce22..2dfd359 100644
--- 
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
+++ 
b/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
@@ -254,10 +254,14 @@ public class LlapTaskCommunicator extends 
TezTaskCommunicatorImpl {
 
   int numHandlers =
   HiveConf.getIntVar(conf, 
ConfVars.LLAP_TASK_COMMUNICATOR_LISTENER_THREAD_COUNT);
+  int umbilicalPort = HiveConf.getIntVar(conf, 
ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT);
+  if (umbilicalPort <= 0) {
+umbilicalPort = 0;
+  }
   server = new RPC.Builder(conf)
   .setProtocol(LlapTaskUmbilicalProtocol.class)
   .setBindAddress("0.0.0.0")
-

[hive] branch master updated: HIVE-21222: ACID: When there are no delete deltas skip finding min max keys (Prasanth Jayachandran reviewed by Eugene Koifman)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 1368bd0  HIVE-21222: ACID: When there are no delete deltas skip 
finding min max keys (Prasanth Jayachandran reviewed by Eugene Koifman)
1368bd0 is described below

commit 1368bd07b5dbc3747390c051e8512a2b41217933
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:22:21 2019 -0800

HIVE-21222: ACID: When there are no delete deltas skip finding min max keys 
(Prasanth Jayachandran reviewed by Eugene Koifman)
---
 .../ql/io/orc/VectorizedOrcAcidRowBatchReader.java |  3 ++-
 .../hive/ql/io/orc/TestInputOutputFormat.java  |  4 +--
 .../orc/TestVectorizedOrcAcidRowBatchReader.java   | 31 --
 3 files changed, 21 insertions(+), 17 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
index 6d1ca722..2349cda 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
@@ -390,7 +390,8 @@ public class VectorizedOrcAcidRowBatchReader
   private OrcRawRecordMerger.KeyInterval findMinMaxKeys(
   OrcSplit orcSplit, Configuration conf,
   Reader.Options deleteEventReaderOptions) throws IOException {
-if(!HiveConf.getBoolVar(conf, ConfVars.FILTER_DELETE_EVENTS)) {
+final boolean noDeleteDeltas = 
getDeleteDeltaDirsFromSplit(orcSplit).length == 0;
+if(!HiveConf.getBoolVar(conf, ConfVars.FILTER_DELETE_EVENTS) || 
noDeleteDeltas) {
   LOG.debug("findMinMaxKeys() " + ConfVars.FILTER_DELETE_EVENTS + 
"=false");
   return new OrcRawRecordMerger.KeyInterval(null, null);
 }
diff --git 
a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
index 50ebbfa..5c13d45 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
@@ -802,8 +802,8 @@ public class TestInputOutputFormat {
   int readsAfter = fs.statistics.getReadOps();
   System.out.println("STATS TRACE END - " + testCaseName.getMethodName());
   int delta = readsAfter - readsBefore;
-  //HIVE-16812 adds 1 read of the footer of each file
-  assertEquals(16, delta);
+  //HIVE-16812 adds 1 read of the footer of each file (only if delete 
delta exists)
+  assertEquals(8, delta);
 } finally {
   MockFileSystem.clearGlobalFiles();
 }
diff --git 
a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 
b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
index 3382288..a8f18d1 100644
--- 
a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
+++ 
b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
@@ -35,6 +35,7 @@ import org.apache.hadoop.hive.ql.exec.Utilities;
 import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
 import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
 import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx;
+import org.apache.hadoop.hive.ql.io.AcidInputFormat;
 import org.apache.hadoop.hive.ql.io.AcidOutputFormat;
 import org.apache.hadoop.hive.ql.io.AcidUtils;
 import org.apache.hadoop.hive.ql.io.BucketCodec;
@@ -60,6 +61,8 @@ import org.junit.Test;
 
 import static org.junit.Assert.*;
 
+import com.google.common.collect.Lists;
+
 /**
  * This class tests the VectorizedOrcAcidRowBatchReader by creating an actual 
split and a set
  * of delete delta files. The split is on an insert delta and there are 
multiple delete deltas
@@ -549,7 +552,7 @@ public class TestVectorizedOrcAcidRowBatchReader {
 OrcSplit split = new OrcSplit(acidFilePath, null,
 stripe.getOffset() + 50,
 stripe.getLength() - 100,
-new String[] {"localhost"}, null, false, true, new ArrayList<>(),
+new String[] {"localhost"}, null, false, true, Lists.newArrayList(new 
AcidInputFormat.DeltaMetaData()),
 fileLength, fileLength, root, null);
 
 validateKeyInterval(split, new RecordIdentifier(1, 1, 1),
@@ -560,7 +563,7 @@ public class TestVectorizedOrcAcidRowBatchReader {
 split = new OrcSplit(acidFilePath, null,
 stripe.getOffset() + 50,
 stripe.getLength() - 100,
-new String[] {"localhost"}, null, false, true, new ArrayList<>(),
+new String[] {"localhost"}, null, false, true, Lists.newArrayList(new 
AcidInputFormat.DeltaMetaData()),
 fileLength, fileLength, root, null);
 
 validateKeyInterval(split, new RecordIdentifier(1, 1, 1),

[hive] branch master updated: HIVE-21235: LLAP: make the name of log4j2 properties file configurable (Prasanth Jayachandran reviewed by Ashuotsh Chauhan)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new cfe2e9b  HIVE-21235: LLAP: make the name of log4j2 properties file 
configurable (Prasanth Jayachandran reviewed by Ashuotsh Chauhan)
cfe2e9b is described below

commit cfe2e9b307cd6c401ee3db2369eaa55e80b2e145
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:20:21 2019 -0800

HIVE-21235: LLAP: make the name of log4j2 properties file configurable 
(Prasanth Jayachandran reviewed by Ashuotsh Chauhan)
---
 llap-server/bin/runLlapDaemon.sh   | 7 ++-
 .../org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java | 1 +
 .../java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java   | 7 +--
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/llap-server/bin/runLlapDaemon.sh b/llap-server/bin/runLlapDaemon.sh
index 30a62f2..69642f4 100755
--- a/llap-server/bin/runLlapDaemon.sh
+++ b/llap-server/bin/runLlapDaemon.sh
@@ -27,6 +27,7 @@ set -x
 #   LLAP_DAEMON_LOGGER - default is console
 #   LLAP_DAEMON_LOG_DIR - defaults to /tmp
 #   LLAP_DAEMON_TMP_DIR - defaults to /tmp
+#   LLAP_LOG4J2_PROPERTIES_FILE_NAME - defaults to 
llap-daemon-log4j2.properties
 #   LLAP_DAEMON_LOG_FILE - 
 #   LLAP_DAEMON_CONF_DIR
 
@@ -90,6 +91,10 @@ if [ "$LLAP_DAEMON_LOGFILE" = "" ]; then
   LLAP_DAEMON_LOG_FILE='llapdaemon.log'
 fi
 
+if [ "LLAP_LOG4J2_PROPERTIES_FILE_NAME" = "" ]; then
+  LLAP_LOG4J2_PROPERTIES_FILE_NAME='llap-daemon-log4j2.properties'
+fi
+
 if [ "$LLAP_DAEMON_HEAPSIZE" = "" ]; then
   LLAP_DAEMON_HEAPSIZE=4096
 fi
@@ -121,7 +126,7 @@ if [ -n "$LLAP_DAEMON_TMP_DIR" ]; then
   export LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Djava.io.tmpdir=$LLAP_DAEMON_TMP_DIR"
 fi
 
-LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dlog4j.configurationFile=llap-daemon-log4j2.properties"
+LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dlog4j.configurationFile=${LLAP_LOG4J2_PROPERTIES_FILE_NAME}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.log.dir=${LLAP_DAEMON_LOG_DIR}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.log.file=${LLAP_DAEMON_LOG_FILE}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.root.logger=${LLAP_DAEMON_LOGGER}"
diff --git 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
index 112173c..b421e0b 100644
--- 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
+++ 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
@@ -19,6 +19,7 @@ import 
org.apache.hadoop.hive.common.classification.InterfaceAudience;
 @InterfaceAudience.Private
 public class LlapConstants {
   public static final String LOG4j2_PROPERTIES_FILE = 
"llap-daemon-log4j2.properties";
+  public static final String LLAP_LOG4J2_PROPERTIES_FILE_NAME_ENV = 
"LLAP_LOG4J2_PROPERTIES_FILE_NAME";
   public static final String LLAP_HADOOP_METRICS2_PROPERTIES_FILE = 
"hadoop-metrics2-llapdaemon.properties";
   public static final String HADOOP_METRICS2_PROPERTIES_FILE = 
"hadoop-metrics2.properties";
 
diff --git 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
index 940be0e..2279f8d 100644
--- 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
+++ 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
@@ -352,8 +352,11 @@ public class LlapDaemon extends CompositeService 
implements ContainerRunner, Lla
 
   private static void initializeLogging(final Configuration conf) {
 long start = System.currentTimeMillis();
-URL llap_l4j2 = LlapDaemon.class.getClassLoader().getResource(
-LlapConstants.LOG4j2_PROPERTIES_FILE);
+String log4j2FileName = 
System.getenv(LlapConstants.LLAP_LOG4J2_PROPERTIES_FILE_NAME_ENV);
+if (log4j2FileName == null || log4j2FileName.isEmpty()) {
+  log4j2FileName = LlapConstants.LOG4j2_PROPERTIES_FILE;
+}
+URL llap_l4j2 = 
LlapDaemon.class.getClassLoader().getResource(log4j2FileName);
 if (llap_l4j2 != null) {
   final boolean async = LogUtils.checkAndSetAsyncLogging(conf);
   // required for MDC based routing appender so that child threads can 
inherit the MDC context

[hive] branch branch-3 updated: HIVE-21235: LLAP: make the name of log4j2 properties file configurable (Prasanth Jayachandran reviewed by Ashuotsh Chauhan)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/branch-3 by this push:
 new 05cedd4  HIVE-21235: LLAP: make the name of log4j2 properties file 
configurable (Prasanth Jayachandran reviewed by Ashuotsh Chauhan)
05cedd4 is described below

commit 05cedd4ae7c0dc00292665590726e343ab76f211
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:20:21 2019 -0800

HIVE-21235: LLAP: make the name of log4j2 properties file configurable 
(Prasanth Jayachandran reviewed by Ashuotsh Chauhan)
---
 llap-server/bin/runLlapDaemon.sh   | 7 ++-
 .../org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java | 1 +
 .../java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java   | 7 +--
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/llap-server/bin/runLlapDaemon.sh b/llap-server/bin/runLlapDaemon.sh
index 1c128c6..ae2621c 100755
--- a/llap-server/bin/runLlapDaemon.sh
+++ b/llap-server/bin/runLlapDaemon.sh
@@ -27,6 +27,7 @@ set -x
 #   LLAP_DAEMON_LOGGER - default is console
 #   LLAP_DAEMON_LOG_DIR - defaults to /tmp
 #   LLAP_DAEMON_TMP_DIR - defaults to /tmp
+#   LLAP_LOG4J2_PROPERTIES_FILE_NAME - defaults to 
llap-daemon-log4j2.properties
 #   LLAP_DAEMON_LOG_FILE - 
 #   LLAP_DAEMON_CONF_DIR
 
@@ -90,6 +91,10 @@ if [ "$LLAP_DAEMON_LOGFILE" = "" ]; then
   LLAP_DAEMON_LOG_FILE='llapdaemon.log'
 fi
 
+if [ "LLAP_LOG4J2_PROPERTIES_FILE_NAME" = "" ]; then
+  LLAP_LOG4J2_PROPERTIES_FILE_NAME='llap-daemon-log4j2.properties'
+fi
+
 if [ "$LLAP_DAEMON_HEAPSIZE" = "" ]; then
   LLAP_DAEMON_HEAPSIZE=4096
 fi
@@ -121,7 +126,7 @@ if [ -n "$LLAP_DAEMON_TMP_DIR" ]; then
   export LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Djava.io.tmpdir=$LLAP_DAEMON_TMP_DIR"
 fi
 
-LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dlog4j.configurationFile=llap-daemon-log4j2.properties"
+LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dlog4j.configurationFile=${LLAP_LOG4J2_PROPERTIES_FILE_NAME}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.log.dir=${LLAP_DAEMON_LOG_DIR}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.log.file=${LLAP_DAEMON_LOG_FILE}"
 LLAP_DAEMON_OPTS="${LLAP_DAEMON_OPTS} 
-Dllap.daemon.root.logger=${LLAP_DAEMON_LOGGER}"
diff --git 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
index 112173c..b421e0b 100644
--- 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
+++ 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapConstants.java
@@ -19,6 +19,7 @@ import 
org.apache.hadoop.hive.common.classification.InterfaceAudience;
 @InterfaceAudience.Private
 public class LlapConstants {
   public static final String LOG4j2_PROPERTIES_FILE = 
"llap-daemon-log4j2.properties";
+  public static final String LLAP_LOG4J2_PROPERTIES_FILE_NAME_ENV = 
"LLAP_LOG4J2_PROPERTIES_FILE_NAME";
   public static final String LLAP_HADOOP_METRICS2_PROPERTIES_FILE = 
"hadoop-metrics2-llapdaemon.properties";
   public static final String HADOOP_METRICS2_PROPERTIES_FILE = 
"hadoop-metrics2.properties";
 
diff --git 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
index 52990c5..b13a2fd 100644
--- 
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
+++ 
b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
@@ -338,8 +338,11 @@ public class LlapDaemon extends CompositeService 
implements ContainerRunner, Lla
 
   private static void initializeLogging(final Configuration conf) {
 long start = System.currentTimeMillis();
-URL llap_l4j2 = LlapDaemon.class.getClassLoader().getResource(
-LlapConstants.LOG4j2_PROPERTIES_FILE);
+String log4j2FileName = 
System.getenv(LlapConstants.LLAP_LOG4J2_PROPERTIES_FILE_NAME_ENV);
+if (log4j2FileName == null || log4j2FileName.isEmpty()) {
+  log4j2FileName = LlapConstants.LOG4j2_PROPERTIES_FILE;
+}
+URL llap_l4j2 = 
LlapDaemon.class.getClassLoader().getResource(log4j2FileName);
 if (llap_l4j2 != null) {
   final boolean async = LogUtils.checkAndSetAsyncLogging(conf);
   // required for MDC based routing appender so that child threads can 
inherit the MDC context

[hive] branch branch-3 updated: HIVE-21223: CachedStore returns null partition when partition does not exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/branch-3 by this push:
 new c0bb1db  HIVE-21223: CachedStore returns null partition when partition 
does not exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)
c0bb1db is described below

commit c0bb1db0bf3fa6853be2777ba20fb9a56a713962
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:18:08 2019 -0800

HIVE-21223: CachedStore returns null partition when partition does not 
exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)
---
 .../main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java   | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
index 2b03d87..39cba08 100644
--- 
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
+++ 
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
@@ -1511,6 +1511,8 @@ public class CachedStore implements RawStore, 
Configurable {
   PrincipalPrivilegeSet privs = getPartitionPrivilegeSet(catName, dbName, 
tblName, partName,
   userName, groupNames);
   p.setPrivileges(privs);
+} else {
+  throw new NoSuchObjectException("partition values=" + 
partVals.toString());
 }
 return p;
   }

[hive] branch master updated: HIVE-21223: CachedStore returns null partition when partition does not exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)

2019-02-12 Thread prasanthj 
This is an automated email from the ASF dual-hosted git repository.

prasanthj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 922b155  HIVE-21223: CachedStore returns null partition when partition 
does not exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)
922b155 is described below

commit 922b155e85e9fdb5bcc27ac5874503483a591021
Author: Prasanth Jayachandran 
AuthorDate: Tue Feb 12 00:18:08 2019 -0800

HIVE-21223: CachedStore returns null partition when partition does not 
exist (Prasanth Jayachandran reviewed by Vaibhav Gumashta)
---
 .../main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java   | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
index 182d5cc..bded743 100644
--- 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
+++ 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
@@ -1884,6 +1884,8 @@ public class CachedStore implements RawStore, 
Configurable {
   PrincipalPrivilegeSet privs = getPartitionPrivilegeSet(catName, dbName, 
tblName, partName,
   userName, groupNames);
   p.setPrivileges(privs);
+} else {
+  throw new NoSuchObjectException("partition values=" + 
partVals.toString());
 }
 return p;
   }