Re: Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems

[Rajesh Balamohan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48159/
---

(Updated June 17, 2016, 5:56 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13901
https://issues.apache.org/jira/browse/HIVE-13901


Repository: hive-git


Description
---

Depending on FS, creating external tables & adding partitions can be expensive 
(e.g msck which adds all partitions).


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc95008 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c0827ea 

Diff: https://reviews.apache.org/r/48159/diff/


Testing
---


Thanks,

Rajesh Balamohan

Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/
---

Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-14029
https://issues.apache.org/jira/browse/HIVE-14029


Repository: hive-git


Description
---

There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark 
to 2.0.0 to benefit those performance improvements.


Diffs
-

  pom.xml 63a5ae1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
 5b65036 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 53c5c0e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
f6595f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java a6350d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
 09c54c1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
ee9f9b7 
  
spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java 
e77aa78 
  spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
e3b88d1 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
 e46b67d 
  spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
a7305cf 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
 be14c06 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
 4420e4d 
  
spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
 5146e91 

Diff: https://reviews.apache.org/r/48839/diff/


Testing
---


Thanks,

cheng xu

[jira] [Created] (HIVE-14043) Add sql support for querying the actual data directory for each partition

[xuchuanyin (JIRA)]

xuchuanyin created HIVE-14043:
-

 Summary: Add sql support for querying the actual data directory 
for each partition
 Key: HIVE-14043
 URL: https://issues.apache.org/jira/browse/HIVE-14043
 Project: Hive
  Issue Type: Wish
Reporter: xuchuanyin
Priority: Trivial


Currently this is no command to query the actual data directory of a partition. 
When this issue solved, we can create a table with its partitions changed day 
by day to map the changing directories.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/
---

(Updated June 17, 2016, 5:44 a.m.)


Review request for hive and Sergey Shelukhin.


Changes
---

Without generated code.


Bugs: HIVE-14023
https://issues.apache.org/jira/browse/HIVE-14023


Repository: hive-git


Description
---

HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner


Diffs (updated)
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 5f250b4 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 01dc2e1 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
85c6091 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
46030ec 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 ded84c1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
a965872 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 6c853a6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 fe2ced5 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 ac48a3a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 fcf3378 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
 3dd73f6 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
 1901328 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
bdf254b 

Diff: https://reviews.apache.org/r/48818/diff/


Testing
---


Thanks,

Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/
---

(Updated June 17, 2016, 5:43 a.m.)


Review request for hive and Sergey Shelukhin.


Bugs: HIVE-14023
https://issues.apache.org/jira/browse/HIVE-14023


Repository: hive-git


Description
---

HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner


Diffs (updated)
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 5f250b4 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 56a1361 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 01dc2e1 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
85c6091 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
46030ec 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 ded84c1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
a965872 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 6c853a6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 fe2ced5 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 ac48a3a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 fcf3378 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
 3dd73f6 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
 1901328 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
bdf254b 

Diff: https://reviews.apache.org/r/48818/diff/


Testing
---


Thanks,

Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

[Siddharth Seth]

> On June 17, 2016, 12:55 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java,
> >  line 436
> > 
> >
> > dag index always 0?

Changed to pick this up from TaskSpec. (that sets it to 0 - the assumption 
being the appId is unique)


> On June 17, 2016, 12:55 a.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java,
> >  line 378
> > 
> >
> > is makeQueryId unique? Should we just use the same thing we use for 
> > generated applicationId? Makes for less things for the user to track.

Supposed to be unique. 
"sseth_20160616180332_06e3b03e-a9ec-4e36-be54-17374c18d339" is an example.
I'll change it to the appId though - single construct does work better.


> On June 17, 2016, 12:55 a.m., Sergey Shelukhin wrote:
> > llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java,
> >  line 269
> > 
> >
> > follow up JIRA? probably under bulk upgrade-to-Tez-v.X one

Will create one.


> On June 17, 2016, 12:55 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java,
> >  line 313
> > 
> >
> > why is this removed? it can be useful for other communication purposes

There's nothing for it to do. It can be re-introduced if required ? (The Hook 
interface still exists in TezProcessor).


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/#review138141
---


On June 16, 2016, 9:16 p.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48818/
> ---
> 
> (Updated June 16, 2016, 9:16 p.m.)
> 
> 
> Review request for hive and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14023
> https://issues.apache.org/jira/browse/HIVE-14023
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  5f250b4 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  56a1361 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> 01dc2e1 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 85c6091 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 46030ec 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  ded84c1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  a965872 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  6c853a6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  fe2ced5 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  ac48a3a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  fcf3378 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
>  3dd73f6 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
>  1901328 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> bdf254b 
> 
> Diff: https://reviews.apache.org/r/48818/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

[jira] [Created] (HIVE-14042) Potential improvements to abort task handling - tez/llap

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14042:
-

 Summary: Potential improvements to abort task handling - tez/llap
 Key: HIVE-14042
 URL: https://issues.apache.org/jira/browse/HIVE-14042
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Follow up from HIVE-14003.

- TezProcessor.run() - in case of an abort, throw an interrupt exception 
instead of falling off. (Falling off cleanly indicates success)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Mohit Sabharwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138081
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 781)


"to the metastore" -> "from the metastore"

"for a given table" instead of "for each partitioned table"



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 1328)


Add the @Deprecated annotation for this config.

For the sake of completeness, we can say "Please use 
ConfVars.METASTORE_LIMIT_PARTITION_REQUEST in the metastore instead."


- Mohit Sabharwal


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

[VOTE] Apache Hive 2.1.0 Release Candidate 3

[Jesus Camacho Rodriguez]

Apache Hive 2.1.0 Release Candidate 3 is available here:

http://people.apache.org/~jcamacho/hive-2.1.0-rc3

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1057/

Source tag for RC3 is at:
https://github.com/apache/hive/releases/tag/release-2.1.0-rc3


Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[jira] [Created] (HIVE-14041) llap scripts add hadoop and other libraries from the machine local install to the daemon classpath

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14041:
-

 Summary: llap scripts add hadoop and other libraries from the 
machine local install to the daemon classpath
 Key: HIVE-14041
 URL: https://issues.apache.org/jira/browse/HIVE-14041
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth


`hadoop classpath` ends up getting added to the classpath of llap daemons. This 
essentially means picking up the classpath from the local deploy.

This isn't required since the slider package includes relevant libraries 
(shipped from the client)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/#review138141
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 


why is this removed? it can be useful for other communication purposes



llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 (line 266)


follow up JIRA? probably under bulk upgrade-to-Tez-v.X one



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
(line 378)


is makeQueryId unique? Should we just use the same thing we use for 
generated applicationId? Makes for less things for the user to track.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
(line 436)


dag index always 0?


- Sergey Shelukhin


On June 16, 2016, 9:16 p.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48818/
> ---
> 
> (Updated June 16, 2016, 9:16 p.m.)
> 
> 
> Review request for hive and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14023
> https://issues.apache.org/jira/browse/HIVE-14023
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
>  5f250b4 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  56a1361 
>   llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 
> 01dc2e1 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
>   llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
> 85c6091 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 46030ec 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  ded84c1 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  a965872 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  6c853a6 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  fe2ced5 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
>  ac48a3a 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  fcf3378 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
>  3dd73f6 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
>  1901328 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> bdf254b 
> 
> Diff: https://reviews.apache.org/r/48818/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

Re: Review Request 48444: HIVE-13913 LLAP: introduce backpressure to recordreader

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48444/
---

(Updated June 17, 2016, 12:35 a.m.)


Review request for hive and Gopal V.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2 
  llap-server/src/java/org/apache/hadoop/hive/llap/ConsumerFeedback.java 
7b2d836 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 298f788 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java
 6b54b30 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 69c0647 

Diff: https://reviews.apache.org/r/48444/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-14040) insert overwrite for HBase doesn't overwrite

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-14040:
---

 Summary: insert overwrite for HBase doesn't overwrite
 Key: HIVE-14040
 URL: https://issues.apache.org/jira/browse/HIVE-14040
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


Creating a table and doing insert overwrite twice with two different rows (for 
example) results in the table with both rows, rather than only one as per 
"overwrite"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Jesus Camacho Rodriguez]

Yes, I think that one is serious enough to stop the RC, thanks for
pointing it out Sushanth.

Since we have only one +1, I think it does not make sense to push this
RC either. I will roll out a new RC shortly and we can restart the vote...

Thanks.




On 6/16/16, 11:47 PM, "Sushanth Sowmyan"  wrote:

>Without HIVE-14020, I'm afraid people will not be able to upgrade the
>hive metastore from an earlier version of hive to 2.1 if they use
>Oracle as a backing db.
>
>There are workarounds, in that the sql script is easily fixed, but
>since we're still in the process of voting a RC, I think this is a big
>enough problem that we should roll out a new RC. I think I'm a -0 on
>this.
>
>
>
>On Thu, Jun 16, 2016 at 2:58 PM, Jesus Camacho Rodriguez
> wrote:
>> Yes, exactly... I am taking care of that once again, do not worry.
>> If you want a precise list of which issues were actually fixed in
>> this release, you can check the release notes in RC2 :)
>>
>>
>>
>>
>> On 6/16/16, 10:32 PM, "Sergey Shelukhin"  wrote:
>>
>>>Hmm… would this mean that all those issues changed from 2.1.1 to 2.1.0
>>>would need to be changed back to 2.1.1 now? ;)
>>>
>>>On 16/6/16, 13:12, "Jesus Camacho Rodriguez"
>>> wrote:
>>>
I have been talking to Matt and HIVE-13974 will not make it to the
release as it needs some
additional time to be fixed. I will add info about this issue to the
release note.

This means RC2 is still alive.

We already got a +1 from Alan. Please, Hive PMC Members, test and vote so
we can move forward
with the release!

Thanks!



On 6/16/16, 11:02 AM, "Jesus Camacho Rodriguez"
 wrote:

>Sure, I am taking care of this each time we roll out a new RC.
>
>
>
>
>On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:
>
>>Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?
>>
>>On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
>> wrote:
>>
>>>OK, vote for RC2 is cancelled.
>>>
>>>Matt, please push HIVE-13974 as soon as possible and I will restart the
>>>vote.
>>>
>>>Thanks,
>>>Jesús
>>>
>>>
>>>
>>>
>>>
>>>On 6/15/16, 9:47 PM, "Matthew McCline" 
>>>wrote:
>>>

-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
non-last STRUCT columns

This bug will prevent people with ORC tables that have added columns
to
inner STRUCT columns to not be able to read their tables.


From: Jesus Camacho Rodriguez 
Sent: Wednesday, June 15, 2016 3:20 AM
To: dev@hive.apache.org
Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

Hive PMC members,

Just a quick reminder that the vote for RC2 is still open and it needs
two additional votes to pass.

Please test and cast your vote!

Thanks,
Jesús



On 6/10/16, 6:29 PM, "Alan Gates"  wrote:

>+1, checked signatures, did a build and ran a few simple unit tests.
>
>Alan.
>
>> On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
>> wrote:
>>
>> Apache Hive 2.1.0 Release Candidate 2 is available here:
>>
>> http://people.apache.org/~jcamacho/hive-2.1.0-rc2
>>
>> Maven artifacts are available here:
>>
>>
>>https://repository.apache.org/content/repositories/orgapachehive-105
>>5/
>>
>> Source tag for RC2 is at:
>> https://github.com/apache/hive/releases/tag/release-2.1.0-rc2
>>
>>
>> Voting will conclude in 72 hours.
>>
>> Hive PMC Members: Please test and vote.
>>
>> Thanks.
>>
>>
>
>

>>
>>>
>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Szehon Ho]

> On June 16, 2016, 9:24 p.m., Szehon Ho wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 780
> > 
> >
> > Should we add this to 'metaVars' variable?  Reading the doc, it seems 
> > it will affect HiveCLI and allow those users to change it on the fly.
> 
> Sergio Pena wrote:
> So 'metaVars' is used to avoid users change it on the fly or to update 
> the metastore when they're changed on the fly? I did not understand the code 
> comment very well.

I think it recreates it for case of embedded metastore.

It is just a suggestion, in case this is the behavior we want for this flag.  
Judging from other flags in this list, seems like it would fit.


- Szehon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138089
---


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Sushanth Sowmyan]

Without HIVE-14020, I'm afraid people will not be able to upgrade the
hive metastore from an earlier version of hive to 2.1 if they use
Oracle as a backing db.

There are workarounds, in that the sql script is easily fixed, but
since we're still in the process of voting a RC, I think this is a big
enough problem that we should roll out a new RC. I think I'm a -0 on
this.



On Thu, Jun 16, 2016 at 2:58 PM, Jesus Camacho Rodriguez
 wrote:
> Yes, exactly... I am taking care of that once again, do not worry.
> If you want a precise list of which issues were actually fixed in
> this release, you can check the release notes in RC2 :)
>
>
>
>
> On 6/16/16, 10:32 PM, "Sergey Shelukhin"  wrote:
>
>>Hmm… would this mean that all those issues changed from 2.1.1 to 2.1.0
>>would need to be changed back to 2.1.1 now? ;)
>>
>>On 16/6/16, 13:12, "Jesus Camacho Rodriguez"
>> wrote:
>>
>>>I have been talking to Matt and HIVE-13974 will not make it to the
>>>release as it needs some
>>>additional time to be fixed. I will add info about this issue to the
>>>release note.
>>>
>>>This means RC2 is still alive.
>>>
>>>We already got a +1 from Alan. Please, Hive PMC Members, test and vote so
>>>we can move forward
>>>with the release!
>>>
>>>Thanks!
>>>
>>>
>>>
>>>On 6/16/16, 11:02 AM, "Jesus Camacho Rodriguez"
>>> wrote:
>>>
Sure, I am taking care of this each time we roll out a new RC.




On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:

>Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?
>
>On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
> wrote:
>
>>OK, vote for RC2 is cancelled.
>>
>>Matt, please push HIVE-13974 as soon as possible and I will restart the
>>vote.
>>
>>Thanks,
>>Jesús
>>
>>
>>
>>
>>
>>On 6/15/16, 9:47 PM, "Matthew McCline" 
>>wrote:
>>
>>>
>>>-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
>>>non-last STRUCT columns
>>>
>>>This bug will prevent people with ORC tables that have added columns
>>>to
>>>inner STRUCT columns to not be able to read their tables.
>>>
>>>
>>>From: Jesus Camacho Rodriguez 
>>>Sent: Wednesday, June 15, 2016 3:20 AM
>>>To: dev@hive.apache.org
>>>Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2
>>>
>>>Hive PMC members,
>>>
>>>Just a quick reminder that the vote for RC2 is still open and it needs
>>>two additional votes to pass.
>>>
>>>Please test and cast your vote!
>>>
>>>Thanks,
>>>Jesús
>>>
>>>
>>>
>>>On 6/10/16, 6:29 PM, "Alan Gates"  wrote:
>>>
+1, checked signatures, did a build and ran a few simple unit tests.

Alan.

> On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
> wrote:
>
> Apache Hive 2.1.0 Release Candidate 2 is available here:
>
> http://people.apache.org/~jcamacho/hive-2.1.0-rc2
>
> Maven artifacts are available here:
>
>
>https://repository.apache.org/content/repositories/orgapachehive-105
>5/
>
> Source tag for RC2 is at:
> https://github.com/apache/hive/releases/tag/release-2.1.0-rc2
>
>
> Voting will conclude in 72 hours.
>
> Hive PMC Members: Please test and vote.
>
> Thanks.
>
>


>>>
>
>>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Sergio Pena]

> On June 16, 2016, 9:24 p.m., Szehon Ho wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 780
> > 
> >
> > Should we add this to 'metaVars' variable?  Reading the doc, it seems 
> > it will affect HiveCLI and allow those users to change it on the fly.

So 'metaVars' is used to avoid users change it on the fly or to update the 
metastore when they're changed on the fly? I did not understand the code 
comment very well.


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138089
---


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

[jira] [Created] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-14039:
---

 Summary: HiveServer2: Make the usage of server with JDBC thirft 
serde enabled, backward compatible for older clients
 Key: HIVE-14039
 URL: https://issues.apache.org/jira/browse/HIVE-14039
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 2.0.1
Reporter: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Jesus Camacho Rodriguez]

Yes, exactly... I am taking care of that once again, do not worry.
If you want a precise list of which issues were actually fixed in
this release, you can check the release notes in RC2 :)




On 6/16/16, 10:32 PM, "Sergey Shelukhin"  wrote:

>Hmm… would this mean that all those issues changed from 2.1.1 to 2.1.0
>would need to be changed back to 2.1.1 now? ;)
>
>On 16/6/16, 13:12, "Jesus Camacho Rodriguez"
> wrote:
>
>>I have been talking to Matt and HIVE-13974 will not make it to the
>>release as it needs some
>>additional time to be fixed. I will add info about this issue to the
>>release note.
>>
>>This means RC2 is still alive.
>>
>>We already got a +1 from Alan. Please, Hive PMC Members, test and vote so
>>we can move forward
>>with the release!
>>
>>Thanks!
>>
>>
>>
>>On 6/16/16, 11:02 AM, "Jesus Camacho Rodriguez"
>> wrote:
>>
>>>Sure, I am taking care of this each time we roll out a new RC.
>>>
>>>
>>>
>>>
>>>On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:
>>>
Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?

On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
 wrote:

>OK, vote for RC2 is cancelled.
>
>Matt, please push HIVE-13974 as soon as possible and I will restart the
>vote.
>
>Thanks,
>Jesús
>
>
>
>
>
>On 6/15/16, 9:47 PM, "Matthew McCline" 
>wrote:
>
>>
>>-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
>>non-last STRUCT columns
>>
>>This bug will prevent people with ORC tables that have added columns
>>to
>>inner STRUCT columns to not be able to read their tables.
>>
>>
>>From: Jesus Camacho Rodriguez 
>>Sent: Wednesday, June 15, 2016 3:20 AM
>>To: dev@hive.apache.org
>>Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2
>>
>>Hive PMC members,
>>
>>Just a quick reminder that the vote for RC2 is still open and it needs
>>two additional votes to pass.
>>
>>Please test and cast your vote!
>>
>>Thanks,
>>Jesús
>>
>>
>>
>>On 6/10/16, 6:29 PM, "Alan Gates"  wrote:
>>
>>>+1, checked signatures, did a build and ran a few simple unit tests.
>>>
>>>Alan.
>>>
 On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
 wrote:

 Apache Hive 2.1.0 Release Candidate 2 is available here:

 http://people.apache.org/~jcamacho/hive-2.1.0-rc2

 Maven artifacts are available here:

 
https://repository.apache.org/content/repositories/orgapachehive-105
5/

 Source tag for RC2 is at:
 https://github.com/apache/hive/releases/tag/release-2.1.0-rc2


 Voting will conclude in 72 hours.

 Hive PMC Members: Please test and vote.

 Thanks.


>>>
>>>
>>

>

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Sergey Shelukhin]

Hmm… would this mean that all those issues changed from 2.1.1 to 2.1.0
would need to be changed back to 2.1.1 now? ;)

On 16/6/16, 13:12, "Jesus Camacho Rodriguez"
 wrote:

>I have been talking to Matt and HIVE-13974 will not make it to the
>release as it needs some
>additional time to be fixed. I will add info about this issue to the
>release note.
>
>This means RC2 is still alive.
>
>We already got a +1 from Alan. Please, Hive PMC Members, test and vote so
>we can move forward
>with the release!
>
>Thanks!
>
>
>
>On 6/16/16, 11:02 AM, "Jesus Camacho Rodriguez"
> wrote:
>
>>Sure, I am taking care of this each time we roll out a new RC.
>>
>>
>>
>>
>>On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:
>>
>>>Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?
>>>
>>>On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
>>> wrote:
>>>
OK, vote for RC2 is cancelled.

Matt, please push HIVE-13974 as soon as possible and I will restart the
vote.

Thanks,
Jesús





On 6/15/16, 9:47 PM, "Matthew McCline" 
wrote:

>
>-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
>non-last STRUCT columns
>
>This bug will prevent people with ORC tables that have added columns
>to
>inner STRUCT columns to not be able to read their tables.
>
>
>From: Jesus Camacho Rodriguez 
>Sent: Wednesday, June 15, 2016 3:20 AM
>To: dev@hive.apache.org
>Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2
>
>Hive PMC members,
>
>Just a quick reminder that the vote for RC2 is still open and it needs
>two additional votes to pass.
>
>Please test and cast your vote!
>
>Thanks,
>Jesús
>
>
>
>On 6/10/16, 6:29 PM, "Alan Gates"  wrote:
>
>>+1, checked signatures, did a build and ran a few simple unit tests.
>>
>>Alan.
>>
>>> On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
>>> wrote:
>>>
>>> Apache Hive 2.1.0 Release Candidate 2 is available here:
>>>
>>> http://people.apache.org/~jcamacho/hive-2.1.0-rc2
>>>
>>> Maven artifacts are available here:
>>>
>>> 
>>>https://repository.apache.org/content/repositories/orgapachehive-105
>>>5/
>>>
>>> Source tag for RC2 is at:
>>> https://github.com/apache/hive/releases/tag/release-2.1.0-rc2
>>>
>>>
>>> Voting will conclude in 72 hours.
>>>
>>> Hive PMC Members: Please test and vote.
>>>
>>> Thanks.
>>>
>>>
>>
>>
>
>>>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Szehon Ho]

> On June 16, 2016, 9:24 p.m., Szehon Ho wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
> > line 4793
> > 
> >
> > Should fix this?

Clarify: fix the name.


- Szehon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138089
---


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Szehon Ho]

> On June 16, 2016, 9:24 p.m., Szehon Ho wrote:
> >

Mostly looks good, just some nits.


- Szehon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138089
---


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/#review138089
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 780)


Should we add this to 'metaVars' variable?  Reading the doc, it seems it 
will affect HiveCLI and allow those users to change it on the fly.



metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (line 
3197)


The "> -1" is not strictly needed as it was already checked earlier by 
isPartitionLimitEnabled.

To be clearer, we should have this method just start with: 

if !isPartitionLimitEnabled() {
  return;
  
that way we don't have to have the extra checks around this method.



metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (line 
4792)


Should fix this?


- Szehon Ho


On June 16, 2016, 4:04 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48233/
> ---
> 
> (Updated June 16, 2016, 4:04 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal and Naveen Gangam.
> 
> 
> Bugs: HIVE-13884
> https://issues.apache.org/jira/browse/HIVE-13884
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch verifies the # of partitions a table has before fetching any from 
> the metastore. I
> t checks that limit from 'hive.limit.query.max.table.partition'.
> 
> A limitation added here is that the variable must be on hive-site.xml in 
> order to work, and it does not accept to set this through beeline because 
> HiveMetaStore.java does not read the variables set through beeline. I think 
> it is better to keep it this way to avoid users changing the value on fly, 
> and crashing the metastore.
> 
> Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
> commands need to fetch partitions in order to create the operator tree. If we 
> allow EXPLAIN to do that, then we may have the same OOM situations for large 
> partitions.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 761dbb279fb196e2bf1e0e59824827a4504eb136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0827ea9d47e569d9697649a7e16d196de3de14d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> c135179b97354108f842a5ca2de0c6f0ef28b7fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> da188d33d6194740ba9ecb37a6e533ecf1ec6906 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
> a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  86a243609b23e2ca9bb8849f0da863a95e477d5c 
> 
> Diff: https://reviews.apache.org/r/48233/diff/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/
---

(Updated June 16, 2016, 9:16 p.m.)


Review request for hive and Sergey Shelukhin.


Summary (updated)
-

HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner


Bugs: HIVE-14023
https://issues.apache.org/jira/browse/HIVE-14023


Repository: hive-git


Description
---

HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 5f250b4 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 56a1361 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 01dc2e1 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
85c6091 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
46030ec 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 ded84c1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
a965872 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 6c853a6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 fe2ced5 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 ac48a3a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 fcf3378 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
 3dd73f6 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
 1901328 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
bdf254b 

Diff: https://reviews.apache.org/r/48818/diff/


Testing
---


Thanks,

Siddharth Seth

Review Request 48818: See jira

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48818/
---

Review request for hive and Sergey Shelukhin.


Bugs: HIVE-14023
https://issues.apache.org/jira/browse/HIVE-14023


Repository: hive-git


Description
---

HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/ext/LlapTaskUmbilicalExternalClient.java
 5f250b4 
  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 56a1361 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java 01dc2e1 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 92dda21 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
85c6091 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
46030ec 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 ded84c1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
a965872 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 6c853a6 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 fe2ced5 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 ac48a3a 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 fcf3378 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/helpers/SourceStateTracker.java
 3dd73f6 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java
 1901328 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
bdf254b 

Diff: https://reviews.apache.org/r/48818/diff/


Testing
---


Thanks,

Siddharth Seth

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Jesus Camacho Rodriguez]

I have been talking to Matt and HIVE-13974 will not make it to the release as 
it needs some
additional time to be fixed. I will add info about this issue to the release 
note.

This means RC2 is still alive.

We already got a +1 from Alan. Please, Hive PMC Members, test and vote so we 
can move forward
with the release!

Thanks!



On 6/16/16, 11:02 AM, "Jesus Camacho Rodriguez" 
 wrote:

>Sure, I am taking care of this each time we roll out a new RC.
>
>
>
>
>On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:
>
>>Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?
>>
>>On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
>> wrote:
>>
>>>OK, vote for RC2 is cancelled.
>>>
>>>Matt, please push HIVE-13974 as soon as possible and I will restart the
>>>vote.
>>>
>>>Thanks,
>>>Jesús
>>>
>>>
>>>
>>>
>>>
>>>On 6/15/16, 9:47 PM, "Matthew McCline"  wrote:
>>>

-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
non-last STRUCT columns

This bug will prevent people with ORC tables that have added columns to
inner STRUCT columns to not be able to read their tables.


From: Jesus Camacho Rodriguez 
Sent: Wednesday, June 15, 2016 3:20 AM
To: dev@hive.apache.org
Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

Hive PMC members,

Just a quick reminder that the vote for RC2 is still open and it needs
two additional votes to pass.

Please test and cast your vote!

Thanks,
Jesús



On 6/10/16, 6:29 PM, "Alan Gates"  wrote:

>+1, checked signatures, did a build and ran a few simple unit tests.
>
>Alan.
>
>> On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
>> wrote:
>>
>> Apache Hive 2.1.0 Release Candidate 2 is available here:
>>
>> http://people.apache.org/~jcamacho/hive-2.1.0-rc2
>>
>> Maven artifacts are available here:
>>
>> https://repository.apache.org/content/repositories/orgapachehive-1055/
>>
>> Source tag for RC2 is at:
>> https://github.com/apache/hive/releases/tag/release-2.1.0-rc2
>>
>>
>> Voting will conclude in 72 hours.
>>
>> Hive PMC Members: Please test and vote.
>>
>> Thanks.
>>
>>
>
>

>>

Review Request 48813: HIVE-13970 refactor LLAPIF splits - get rid of SubmitWorkInfo

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48813/
---

Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

.


Diffs
-

  llap-client/src/java/org/apache/hadoop/hive/llap/LlapInputSplit.java 2896651 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 95b0ffc 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
46030ec 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
bdf254b 

Diff: https://reviews.apache.org/r/48813/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-14038) miscellaneous acid improvements

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-14038:
-

 Summary: miscellaneous acid improvements
 Key: HIVE-14038
 URL: https://issues.apache.org/jira/browse/HIVE-14038
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


1. fix thread name inHouseKeeperServiceBase
2. dump metastore configs from HiveConf on start up to help record values of 
properties
3. add some tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-14037:
---

 Summary: java.lang.ClassNotFoundException for the jar in 
hive.reloadable.aux.jars.path in mapreduce 
 Key: HIVE-14037
 URL: https://issues.apache.org/jira/browse/HIVE-14037
 Project: Hive
  Issue Type: Bug
Reporter: Aihua Xu
Assignee: Aihua Xu


The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process 
while they are not available in the Mapper or Reducer nodes which will throw 
the following exception.

{noformat}
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
... 22 more
Caused by: java.lang.ClassNotFoundException: test.UDF
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
... 36 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14036) prevent insert into acid and non-acid tables in the same txn

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-14036:
-

 Summary: prevent insert into acid and non-acid tables in the same 
txn
 Key: HIVE-14036
 URL: https://issues.apache.org/jira/browse/HIVE-14036
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.0.0, 1.2.0
Reporter: Eugene Koifman


{noformat}
create table if not exists tab1 (a int, b int) partitioned by (p string) 
clustered by (a) into 2  buckets stored as orc 
TBLPROPERTIES('transactional'='true')
create table if not exists tab_not_acid (a int, b int, p string)
insert into tab_not_acid values(1,1,'one'),(2,2,'two')

insert into tab1 partition(p) values(3,3,'one'),(4,4,'two')
//writing both acid and non-acid resources in the same txn
//tab1 write is a dynamic partition insert
from tab_not_acid insert into tab1 partition(p)(a,b,p) select a,b,p insert 
into tab_not_acid(a,b) select a,b where p='two'
{noformat}

Suppose the last insert in the above example is part of a multi-stmt txn which 
later gets rolled back.  There is no way to rollback the write to the 
tab_not_acid table.

(In auto-commit mode there is no explicit rollback so this can only fail due to 
error, which means Move task won't move the files into tables so it's not 
really an issue)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Assign JIRA to myself

[Thejas Nair]

This is now fixed again via the infra ticket, you don't need to be in
contributor list anymore to be able to assign yourself jiras.


On Thu, Jun 16, 2016 at 8:48 AM, Thejas Nair  wrote:
> created https://issues.apache.org/jira/browse/INFRA-12111
>
> On Thu, Jun 16, 2016 at 8:44 AM, Thejas Nair  wrote:
>> We had an earlier discussion in email thread about how having to
>> request to be a contributor to be able to assign themselves a jira is
>> an unnecessary bureaucratic step. The way I had made that change was
>> to add jira-users to contributors list. I think that change must have
>> been reverted because of the spam issues we have been having in jira.
>>
>> The better way to grant the permissions would be to use JIRA
>> permissions page, but I don't see a way to do that. I will get in
>> touch with INFRA if they can grant the necessary privilges to jira
>> admins to be able to do so.
>>
>>
>> On Thu, Jun 16, 2016 at 8:18 AM, Peter Vary  wrote:
>>> Thanks Ashutosh!
>>>
>>> Checked, and OK.
>>>
>>> Peter
>>>
 On Jun 16, 2016, at 4:24 PM, Ashutosh Chauhan  wrote:

 Hi Peter,
 I have added you in a contributor list. Now it should be possible for you
 to assign jira to yourself.

 Thanks,
 Ashutosh

 On Wed, Jun 15, 2016 at 11:22 PM, Peter Vary  wrote:

> Hi,
>
> Thanks for the fast answer.
> Who should I ask to add me as a contributor? 6 days ago I sent here a
> request, but have not got any reply.
>
> Thanks,
> Péter
> 2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):
>
>> Hi Peter,
>>
>> You’re not added as a contributor. Please see additional information at
>>
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>>
>>
>>
>> *From:* Peter Vary [mailto:pv...@cloudera.com]
>> *Sent:* Thursday, June 16, 2016 12:15 AM
>> *To:* dev@hive.apache.org
>> *Subject:* Assign JIRA to myself
>>
>>
>>
>> Hi everyone,
>>
>>
>>
>> I am trying to assign a JIRA to myself, but could not (see: screenshot).
>>
>> Can anyone help me there?
>>
>>
>>
>> Thanks in advance,
>>
>> Peter
>>
>>
>>
>>
>
>>>

[jira] [Created] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions

[Saket Saurabh (JIRA)]

Saket Saurabh created HIVE-14035:


 Summary: Enable predicate pushdown to delta files created by ACID 
Transactions
 Key: HIVE-14035
 URL: https://issues.apache.org/jira/browse/HIVE-14035
 Project: Hive
  Issue Type: New Feature
  Components: Transactions
Reporter: Saket Saurabh
Priority: Minor


In current Hive version, delta files created by ACID transactions do not allow 
predicate pushdown if they contain any update/delete events. This is done to 
preserve correctness when following a multi-version approach during event 
collapsing, where an update event overwrites an existing insert event. 
This JIRA proposes to split an update event into a combination of a delete 
event followed by a new insert event, that can enable predicate push down to 
all delta files without breaking correctness. To support backward compatibility 
for this feature, this JIRA also proposes to add some sort of versioning to 
ACID that can allow different versions of ACID transactions to co-exist 
together.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14034) Vectorization may fail with compex OR conditions

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-14034:
---

 Summary: Vectorization may fail with compex OR conditions
 Key: HIVE-14034
 URL: https://issues.apache.org/jira/browse/HIVE-14034
 Project: Hive
  Issue Type: Bug
Reporter: Takahiko Saito
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 48798: HIVE-14001: beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48798/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-14001
https://issues.apache.org/jira/browse/HIVE-14001


Repository: hive-git


Description
---

When providing both arguments there should be an error message


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 734eeb8 
  beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java 80c6e06 

Diff: https://reviews.apache.org/r/48798/diff/


Testing
---

Unittest with both arguments present.
Manual testing


Thanks,

Peter Vary

[jira] [Created] (HIVE-14033) Hive converts timezone of device executing job

[Dedric Friedly (JIRA)]

Dedric Friedly created HIVE-14033:
-

 Summary: Hive converts timezone of device executing job
 Key: HIVE-14033
 URL: https://issues.apache.org/jira/browse/HIVE-14033
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.1
Reporter: Dedric Friedly


I've noticed an issue where Hive will "silently" convert timezones based on the 
local devices timezone that's executing the query.

This will cause issues if nodes are on different timezones in the cluster.

An example of the issue can be found here:  
http://stackoverflow.com/questions/37864004/hive-returning-wrong-date/37864658#37864658



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48771: HIVE-13590: Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case

[Chaoyu Tang]

> On June 16, 2016, 3:01 p.m., Sergio Pena wrote:
> > The code looks good for me.
> > 
> > What other tests did you do to validate this code?
> > - HS2 with KERBEROS + LDAP?
> > - HS2 with KERBEROS only?
> > - HS2 with LDAP only?

The unit tests cover the cases:
1. HS2 with kerberos + SASL non-kerberos authentication. I used the CUSTOM 
authentication since Hive does not have LDAP end-to-end unit test framework and 
CUSTOM authentication shares the same code path with LDAP except the detail how 
LDAP does its authentication, which is not the scope of this JIRA. So in term 
of this JIRA, testing with LDAP and CUSTOM authentication is equalivent and 
actually LDAP is a specially case of CUSTOM.
2. HS2 with kerberos only
3. HS2 with only SASL non-kerberos authentication (CUSTOM) for same reason as 
item 1.


> On June 16, 2016, 3:01 p.m., Sergio Pena wrote:
> > service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java, line 231
> > 
> >
> > Should we return empty strings instead of nulls? That way we avoid that 
> > any other future developer uses this method without validating a null 
> > pointer, that could cause NPE if they do not do that.

Thanks Sergio for review. yeah, it is debatable which way is better, returning 
null or empty object (e.g. String), please see 
http://stackoverflow.com/questions/1626597/should-functions-return-null-or-an-empty-object
Thinking over, I tend to still return null for this API because
1. There is not an "empty" authentication mechanism. It either exists like 
TOKEN, KERBEROS, or does not exist (which null will be more informative)
2. Be consistent with other APIs like getRemoteUser(), getIpAddress()

I wonder if it makes sense. Thanks


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48771/#review137988
---


On June 16, 2016, 2:33 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48771/
> ---
> 
> (Updated June 16, 2016, 2:33 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13590
> https://issues.apache.org/jira/browse/HIVE-13590
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive should not use Hadoop security (e.g. kerberos) related APIs such as 
> KerberosName etc to process user logged in via other SASL mechanism such as 
> LDAP.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcNonKrbSASLWithMiniKdc.java
>  1c1beda 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java ab8806c 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8bc3d94 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
>  8a4786c 
> 
> Diff: https://reviews.apache.org/r/48771/diff/
> 
> 
> Testing
> ---
> 
> Manual test
> PreCommit test
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 48233: HIVE-13884: Disallow queries fetching more than a configured number of partitions in PartitionPruner

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48233/
---

(Updated June 16, 2016, 4:04 p.m.)


Review request for hive, Mohit Sabharwal and Naveen Gangam.


Bugs: HIVE-13884
https://issues.apache.org/jira/browse/HIVE-13884


Repository: hive-git


Description
---

The patch verifies the # of partitions a table has before fetching any from the 
metastore. I
t checks that limit from 'hive.limit.query.max.table.partition'.

A limitation added here is that the variable must be on hive-site.xml in order 
to work, and it does not accept to set this through beeline because 
HiveMetaStore.java does not read the variables set through beeline. I think it 
is better to keep it this way to avoid users changing the value on fly, and 
crashing the metastore.

Another change is that EXPLAIN commands won't be executed either. EXPLAIN 
commands need to fetch partitions in order to create the operator tree. If we 
allow EXPLAIN to do that, then we may have the same OOM situations for large 
partitions.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
761dbb279fb196e2bf1e0e59824827a4504eb136 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c0827ea9d47e569d9697649a7e16d196de3de14d 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
c135179b97354108f842a5ca2de0c6f0ef28b7fc 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
da188d33d6194740ba9ecb37a6e533ecf1ec6906 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 
a6d3f5385b33b8a4e31ee20ca5cb8f58c97c8702 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
31f0d7b89670b8a749bbe8a7ff2b4ff9f059a8e2 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 3152e77c3c7152ac4dbe7e779ce35f28044fe3c9 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 86a243609b23e2ca9bb8849f0da863a95e477d5c 

Diff: https://reviews.apache.org/r/48233/diff/


Testing
---

Waiting for HiveQA.


Thanks,

Sergio Pena

Re: Assign JIRA to myself

[Thejas Nair]

created https://issues.apache.org/jira/browse/INFRA-12111

On Thu, Jun 16, 2016 at 8:44 AM, Thejas Nair  wrote:
> We had an earlier discussion in email thread about how having to
> request to be a contributor to be able to assign themselves a jira is
> an unnecessary bureaucratic step. The way I had made that change was
> to add jira-users to contributors list. I think that change must have
> been reverted because of the spam issues we have been having in jira.
>
> The better way to grant the permissions would be to use JIRA
> permissions page, but I don't see a way to do that. I will get in
> touch with INFRA if they can grant the necessary privilges to jira
> admins to be able to do so.
>
>
> On Thu, Jun 16, 2016 at 8:18 AM, Peter Vary  wrote:
>> Thanks Ashutosh!
>>
>> Checked, and OK.
>>
>> Peter
>>
>>> On Jun 16, 2016, at 4:24 PM, Ashutosh Chauhan  wrote:
>>>
>>> Hi Peter,
>>> I have added you in a contributor list. Now it should be possible for you
>>> to assign jira to yourself.
>>>
>>> Thanks,
>>> Ashutosh
>>>
>>> On Wed, Jun 15, 2016 at 11:22 PM, Peter Vary  wrote:
>>>
 Hi,

 Thanks for the fast answer.
 Who should I ask to add me as a contributor? 6 days ago I sent here a
 request, but have not got any reply.

 Thanks,
 Péter
 2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):

> Hi Peter,
>
> You’re not added as a contributor. Please see additional information at
>
 https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>
>
>
> *From:* Peter Vary [mailto:pv...@cloudera.com]
> *Sent:* Thursday, June 16, 2016 12:15 AM
> *To:* dev@hive.apache.org
> *Subject:* Assign JIRA to myself
>
>
>
> Hi everyone,
>
>
>
> I am trying to assign a JIRA to myself, but could not (see: screenshot).
>
> Can anyone help me there?
>
>
>
> Thanks in advance,
>
> Peter
>
>
>
>

>>

Re: Assign JIRA to myself

[Thejas Nair]

We had an earlier discussion in email thread about how having to
request to be a contributor to be able to assign themselves a jira is
an unnecessary bureaucratic step. The way I had made that change was
to add jira-users to contributors list. I think that change must have
been reverted because of the spam issues we have been having in jira.

The better way to grant the permissions would be to use JIRA
permissions page, but I don't see a way to do that. I will get in
touch with INFRA if they can grant the necessary privilges to jira
admins to be able to do so.


On Thu, Jun 16, 2016 at 8:18 AM, Peter Vary  wrote:
> Thanks Ashutosh!
>
> Checked, and OK.
>
> Peter
>
>> On Jun 16, 2016, at 4:24 PM, Ashutosh Chauhan  wrote:
>>
>> Hi Peter,
>> I have added you in a contributor list. Now it should be possible for you
>> to assign jira to yourself.
>>
>> Thanks,
>> Ashutosh
>>
>> On Wed, Jun 15, 2016 at 11:22 PM, Peter Vary  wrote:
>>
>>> Hi,
>>>
>>> Thanks for the fast answer.
>>> Who should I ask to add me as a contributor? 6 days ago I sent here a
>>> request, but have not got any reply.
>>>
>>> Thanks,
>>> Péter
>>> 2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):
>>>
 Hi Peter,

 You’re not added as a contributor. Please see additional information at

>>> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor



 *From:* Peter Vary [mailto:pv...@cloudera.com]
 *Sent:* Thursday, June 16, 2016 12:15 AM
 *To:* dev@hive.apache.org
 *Subject:* Assign JIRA to myself



 Hi everyone,



 I am trying to assign a JIRA to myself, but could not (see: screenshot).

 Can anyone help me there?



 Thanks in advance,

 Peter




>>>
>

Re: Assign JIRA to myself

[Peter Vary]

Thanks Ashutosh!

Checked, and OK.

Peter

> On Jun 16, 2016, at 4:24 PM, Ashutosh Chauhan  wrote:
> 
> Hi Peter,
> I have added you in a contributor list. Now it should be possible for you
> to assign jira to yourself.
> 
> Thanks,
> Ashutosh
> 
> On Wed, Jun 15, 2016 at 11:22 PM, Peter Vary  wrote:
> 
>> Hi,
>> 
>> Thanks for the fast answer.
>> Who should I ask to add me as a contributor? 6 days ago I sent here a
>> request, but have not got any reply.
>> 
>> Thanks,
>> Péter
>> 2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):
>> 
>>> Hi Peter,
>>> 
>>> You’re not added as a contributor. Please see additional information at
>>> 
>> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>>> 
>>> 
>>> 
>>> *From:* Peter Vary [mailto:pv...@cloudera.com]
>>> *Sent:* Thursday, June 16, 2016 12:15 AM
>>> *To:* dev@hive.apache.org
>>> *Subject:* Assign JIRA to myself
>>> 
>>> 
>>> 
>>> Hi everyone,
>>> 
>>> 
>>> 
>>> I am trying to assign a JIRA to myself, but could not (see: screenshot).
>>> 
>>> Can anyone help me there?
>>> 
>>> 
>>> 
>>> Thanks in advance,
>>> 
>>> Peter
>>> 
>>> 
>>> 
>>> 
>> 

[jira] [Created] (HIVE-14032) INSERT OVERWRITE command failed with sensitive names.

[Chinna Rao Lalam (JIRA)]

Chinna Rao Lalam created HIVE-14032:
---

 Summary: INSERT OVERWRITE command failed with sensitive names.
 Key: HIVE-14032
 URL: https://issues.apache.org/jira/browse/HIVE-14032
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.0.1
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48771: HIVE-13590: Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48771/#review137988
---


Fix it, then Ship it!




The code looks good for me.

What other tests did you do to validate this code?
- HS2 with KERBEROS + LDAP?
- HS2 with KERBEROS only?
- HS2 with LDAP only?


service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java (line 231)


Should we return empty strings instead of nulls? That way we avoid that any 
other future developer uses this method without validating a null pointer, that 
could cause NPE if they do not do that.


- Sergio Pena


On June 16, 2016, 2:33 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48771/
> ---
> 
> (Updated June 16, 2016, 2:33 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13590
> https://issues.apache.org/jira/browse/HIVE-13590
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive should not use Hadoop security (e.g. kerberos) related APIs such as 
> KerberosName etc to process user logged in via other SASL mechanism such as 
> LDAP.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcNonKrbSASLWithMiniKdc.java
>  1c1beda 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java ab8806c 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8bc3d94 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
>  8a4786c 
> 
> Diff: https://reviews.apache.org/r/48771/diff/
> 
> 
> Testing
> ---
> 
> Manual test
> PreCommit test
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Assign JIRA to myself

[Ashutosh Chauhan]

Hi Peter,
I have added you in a contributor list. Now it should be possible for you
to assign jira to yourself.

Thanks,
Ashutosh

On Wed, Jun 15, 2016 at 11:22 PM, Peter Vary  wrote:

> Hi,
>
> Thanks for the fast answer.
> Who should I ask to add me as a contributor? 6 days ago I sent here a
> request, but have not got any reply.
>
> Thanks,
> Péter
> 2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):
>
> > Hi Peter,
> >
> > You’re not added as a contributor. Please see additional information at
> >
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
> >
> >
> >
> > *From:* Peter Vary [mailto:pv...@cloudera.com]
> > *Sent:* Thursday, June 16, 2016 12:15 AM
> > *To:* dev@hive.apache.org
> > *Subject:* Assign JIRA to myself
> >
> >
> >
> > Hi everyone,
> >
> >
> >
> > I am trying to assign a JIRA to myself, but could not (see: screenshot).
> >
> > Can anyone help me there?
> >
> >
> >
> > Thanks in advance,
> >
> > Peter
> >
> >
> >
> >
>

Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2

[Jesus Camacho Rodriguez]

Sure, I am taking care of this each time we roll out a new RC.




On 6/15/16, 10:43 PM, "Sergey Shelukhin"  wrote:

>Should all the 2.1.1-fixed JIRAs be converted to 2.1.0?
>
>On 16/6/15, 14:03, "Jesus Camacho Rodriguez"
> wrote:
>
>>OK, vote for RC2 is cancelled.
>>
>>Matt, please push HIVE-13974 as soon as possible and I will restart the
>>vote.
>>
>>Thanks,
>>Jesús
>>
>>
>>
>>
>>
>>On 6/15/16, 9:47 PM, "Matthew McCline"  wrote:
>>
>>>
>>>-1 for HIVE-13974 ORC Schema Evolution doesn't support add columns to
>>>non-last STRUCT columns
>>>
>>>This bug will prevent people with ORC tables that have added columns to
>>>inner STRUCT columns to not be able to read their tables.
>>>
>>>
>>>From: Jesus Camacho Rodriguez 
>>>Sent: Wednesday, June 15, 2016 3:20 AM
>>>To: dev@hive.apache.org
>>>Subject: Re: [VOTE] Apache Hive 2.1.0 Release Candidate 2
>>>
>>>Hive PMC members,
>>>
>>>Just a quick reminder that the vote for RC2 is still open and it needs
>>>two additional votes to pass.
>>>
>>>Please test and cast your vote!
>>>
>>>Thanks,
>>>Jesús
>>>
>>>
>>>
>>>On 6/10/16, 6:29 PM, "Alan Gates"  wrote:
>>>
+1, checked signatures, did a build and ran a few simple unit tests.

Alan.

> On Jun 10, 2016, at 05:44, Jesus Camacho Rodriguez
> wrote:
>
> Apache Hive 2.1.0 Release Candidate 2 is available here:
>
> http://people.apache.org/~jcamacho/hive-2.1.0-rc2
>
> Maven artifacts are available here:
>
> https://repository.apache.org/content/repositories/orgapachehive-1055/
>
> Source tag for RC2 is at:
> https://github.com/apache/hive/releases/tag/release-2.1.0-rc2
>
>
> Voting will conclude in 72 hours.
>
> Hive PMC Members: Please test and vote.
>
> Thanks.
>
>


>>>
>

RE: Assign JIRA to myself

[Peter Vary]

Hi,

Thanks for the fast answer.
Who should I ask to add me as a contributor? 6 days ago I sent here a
request, but have not got any reply.

Thanks,
Péter
2016. jún. 16. 3:13 ezt írta ("Xu, Cheng A" ):

> Hi Peter,
>
> You’re not added as a contributor. Please see additional information at
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>
>
>
> *From:* Peter Vary [mailto:pv...@cloudera.com]
> *Sent:* Thursday, June 16, 2016 12:15 AM
> *To:* dev@hive.apache.org
> *Subject:* Assign JIRA to myself
>
>
>
> Hi everyone,
>
>
>
> I am trying to assign a JIRA to myself, but could not (see: screenshot).
>
> Can anyone help me there?
>
>
>
> Thanks in advance,
>
> Peter
>
>
>
>

[jira] [Created] (HIVE-14031) cleanup metadataReader in OrcEncodedDataReader

[Rajesh Balamohan (JIRA)]

Rajesh Balamohan created HIVE-14031:
---

 Summary: cleanup metadataReader in OrcEncodedDataReader
 Key: HIVE-14031
 URL: https://issues.apache.org/jira/browse/HIVE-14031
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan
Priority: Minor


MetadataReader should be closed in OrcEncodedDataReader as a part of 
cleanupReaders. 

\cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14030) LLAP: implement LLAP UI as a separate service - part 2 - multi llap nodes

[Yuta Imai (JIRA)]

Yuta Imai created HIVE-14030:


 Summary: LLAP: implement LLAP UI as a separate service - part 2 - 
multi llap nodes
 Key: HIVE-14030
 URL: https://issues.apache.org/jira/browse/HIVE-14030
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Yuta Imai


So far we have done single node LLAP UI on HIVE-11526. We are going to add 
multi nodes capability for this UI so we can have a cluster wide metrics at a 
glance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)