Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review208006
---



Very useful feature, thanks for doing this! A bunch of nits below.


standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1645 (patched)


Can you put top-level comment for this struct explaining that this is an 
API to control both partition filtering and partition content.



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1647 (patched)


I think it is better to specify that API guarantees that it will include 
fields in the list although it may set some other fields as well - the contract 
is that *these fields* must be set.



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1654 (patched)


s/complaint/compliant/



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1656 (patched)


Can you add some statement at the top telling that the API allows for param 
filtering as well.



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1659 (patched)


Can you specify what happens if some match is both included and excluded?



standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
Lines 1668 (patched)


Do fields 1-6 belong here or in GetPartitionRequest? Essentially - are 
these part of the filter spec or part of the request asking for partitions?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2612 (patched)


This wouldn't be a very useful MPart - will something set correct dbname, 
etc later?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 3358 (patched)


It would be good to add a comment that this is JDO-only code.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 3406 (patched)


mparts can be a huge list of objects, so printing this in debug log may be 
too much - may be just log number of objects?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 4035 (patched)


What about filtering params? Does it still apply?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 4051 (patched)


Nit: comments have spce after //



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 4052 (patched)


in setRange() or setResults()?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 4393 (original), 4508 (patched)


Is this legit change?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Line 547 (original), 553 (patched)


Please document this API.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Lines 554 (patched)


I don't think that `allowJDO` and `allowSql` belong here - this is an API 
for the raw store and jdo is an implementation detail of specific RawStore. So 
this should be an implementation that deals with it (probaby based on config or 
hardcore values) but we shouldn't be exposing this in the interface.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Lines 555 (patched)


What if we later extern projectionSpec or filters to include more things? 
Would we need to add new methods here? Can we just pass either original Thrift 
request structs or some some representation of these so that we can extend 
things later without introducing new API methods?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Lines 556 (patched)


When would it return

Review Request 68534: HIVE-20432 Rewrite BETWEEN to IN for integer types for stats estimation

2018-08-27 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68534/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20432
https://issues.apache.org/jira/browse/HIVE-20432


Repository: hive-git


Description
---

See JIRA description


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9cd6812ce4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 5275561619 
  ql/src/test/results/clientpositive/join46.q.out c192194329 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
af7b0ba2ba 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
84c5b2ff5e 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 83a94f8856 
  ql/src/test/results/clientpositive/mapjoin46.q.out a4f067f101 
  ql/src/test/results/clientpositive/perf/spark/query10.q.out b7faa9a130 
  ql/src/test/results/clientpositive/perf/spark/query22.q.out d0970bba16 
  ql/src/test/results/clientpositive/perf/spark/query25.q.out 3d7f402870 
  ql/src/test/results/clientpositive/perf/spark/query28.q.out b437829c70 
  ql/src/test/results/clientpositive/perf/spark/query29.q.out a7347102c1 
  ql/src/test/results/clientpositive/perf/spark/query34.q.out 88279a30a7 
  ql/src/test/results/clientpositive/perf/spark/query38.q.out 0064177f96 
  ql/src/test/results/clientpositive/perf/spark/query51.q.out 21afbe2024 
  ql/src/test/results/clientpositive/perf/spark/query59.q.out 1224ab68a6 
  ql/src/test/results/clientpositive/perf/spark/query65.q.out 4afa1a 
  ql/src/test/results/clientpositive/perf/spark/query67.q.out db91e67fdc 
  ql/src/test/results/clientpositive/perf/spark/query68.q.out f2e676381f 
  ql/src/test/results/clientpositive/perf/spark/query69.q.out e17832c8e8 
  ql/src/test/results/clientpositive/perf/spark/query70.q.out 4222b5262d 
  ql/src/test/results/clientpositive/perf/spark/query73.q.out ddaca3d4a1 
  ql/src/test/results/clientpositive/perf/spark/query79.q.out a83090f27c 
  ql/src/test/results/clientpositive/perf/spark/query86.q.out 1d1e4ef3d3 
  ql/src/test/results/clientpositive/perf/spark/query87.q.out 8ac6dcedbe 
  ql/src/test/results/clientpositive/perf/spark/query9.q.out a43450147e 
  ql/src/test/results/clientpositive/perf/spark/query90.q.out 9f78d64bfb 
  ql/src/test/results/clientpositive/perf/spark/query97.q.out c4f4804452 
  ql/src/test/results/clientpositive/perf/spark/query99.q.out f541da81a6 
  ql/src/test/results/clientpositive/perf/tez/query10.q.out 5b55d44d91 
  ql/src/test/results/clientpositive/perf/tez/query14.q.out b9efa45853 
  ql/src/test/results/clientpositive/perf/tez/query22.q.out cfe780a270 
  ql/src/test/results/clientpositive/perf/tez/query25.q.out b68c54aef7 
  ql/src/test/results/clientpositive/perf/tez/query28.q.out ca0bf37abf 
  ql/src/test/results/clientpositive/perf/tez/query29.q.out b05162282c 
  ql/src/test/results/clientpositive/perf/tez/query34.q.out 7d140921d9 
  ql/src/test/results/clientpositive/perf/tez/query38.q.out 78cb5e15cb 
  ql/src/test/results/clientpositive/perf/tez/query51.q.out ba3f994b42 
  ql/src/test/results/clientpositive/perf/tez/query59.q.out a4be291b1b 
  ql/src/test/results/clientpositive/perf/tez/query65.q.out c9759c396d 
  ql/src/test/results/clientpositive/perf/tez/query67.q.out 960e66861c 
  ql/src/test/results/clientpositive/perf/tez/query68.q.out fd1e04b328 
  ql/src/test/results/clientpositive/perf/tez/query69.q.out 4aad667ba3 
  ql/src/test/results/clientpositive/perf/tez/query70.q.out 4efb85a52c 
  ql/src/test/results/clientpositive/perf/tez/query73.q.out b18b8608db 
  ql/src/test/results/clientpositive/perf/tez/query79.q.out 7b9d48f123 
  ql/src/test/results/clientpositive/perf/tez/query86.q.out 8140b72795 
  ql/src/test/results/clientpositive/perf/tez/query87.q.out 1b9256d53e 
  ql/src/test/results/clientpositive/perf/tez/query9.q.out 611daa8b88 
  ql/src/test/results/clientpositive/perf/tez/query90.q.out ae5c596192 
  ql/src/test/results/clientpositive/perf/tez/query97.q.out b65b57aac1 
  ql/src/test/results/clientpositive/perf/tez/query99.q.out 456fd8c96f 
  ql/src/test/results/clientpositive/smb_mapjoin_46.q.out 1627af2561 
  ql/src/test/results/clientpositive/smb_mapjoin_47.q.out 9cc2de467f 
  
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_3.q.out
 a1414090f3 
  ql/src/test/results/clientpositive/spark/subquery_select.q.out dc7a510510 


Diff: https://reviews.apache.org/r/68534/diff/1/


Testing
---


Thanks,

Vineet Garg

[jira] [Created] (HIVE-20472) mvn test failing for metastore-tool module

2018-08-27 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-20472:
--

 Summary: mvn test failing for metastore-tool module
 Key: HIVE-20472
 URL: https://issues.apache.org/jira/browse/HIVE-20472
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Fails because there are no applicable tests.

 

{code}

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on 
project hive-metastore-benchmarks: No tests were executed! (Set 
-DfailIfNoTests=false to ignore this error.) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn  -rf :hive-metastore-benchmarks

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Vihang Karajgaonkar via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 27, 2018, 10:33 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Added support for JDO result for projected fields.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 47f96f323ade4e15753d6fd98709b9a882de624d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 3c0d0a55b1dcca96dca77676873b68e52703715d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 e54a7321e2e1e6069a3e598627bc6f6eaed93449 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
feb44d5159f131fae932739923b1a41f5e72e74b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 33b22a9fc3e60cb6e11bec63d397f1fa712a41db 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a0ff79cc5ca39f8af8bac672393e82d365c9fd4a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
 4c6ce008f89469353bfee3175168a518534a42b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 10ff9dfbb6d8f61fa75f731f4cd0f006c98e0067 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 c681a87a1c6b10a4f9494e49a42282cf90027ad7

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Vihang Karajgaonkar via Review Board 


> On Aug. 27, 2018, 4:42 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
> > Line 200 (original), 200 (patched)
> > 
> >
> > Seems it can be static since it doesn't refer to any variables outside. 
> > Maybe I miss anything?

changed this method to static as suggested


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207981
---


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
>

[jira] [Created] (HIVE-20471) issues getting the default database path

2018-08-27 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20471:
---

 Summary: issues getting the default database path
 Key: HIVE-20471
 URL: https://issues.apache.org/jira/browse/HIVE-20471
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


{noformat}
Caused by: java.lang.IllegalArgumentException: Can not create a Path from an 
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)
at org.apache.hadoop.fs.Path.(Path.java:237)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:143)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:147)
at 
org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:160)
at 
org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:235)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2335)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2074)

{noformat}

given that the path passed to getDnsPath is already a path that is probably 
valid, this method needs to be a little bit more reliable. Not sure which 
argument turns out to be empty, but I wonder why this complexity is needed for 
the default database.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20470) Runtime of QueryInfo should be initialized

2018-08-27 Thread Alice Fan (JIRA) 
Alice Fan created HIVE-20470:


 Summary: Runtime of QueryInfo should be initialized
 Key: HIVE-20470
 URL: https://issues.apache.org/jira/browse/HIVE-20470
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Alice Fan
Assignee: Alice Fan
 Fix For: 4.0.0


Runtime of QueryInfo should be initialized to prevent NPE at hiveserver2_jsp



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20469) Do not rollup PK/FK columns when indexing to druid.

2018-08-27 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20469:
---

 Summary: Do not rollup PK/FK columns when indexing to druid. 
 Key: HIVE-20469
 URL: https://issues.apache.org/jira/browse/HIVE-20469
 Project: Hive
  Issue Type: Bug
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


When indexing data to druid if a numeric column has a PK/FK constraint. 
We need to make sure it is not indexed as a metric and rolled up when indexing 
to druid. 

Thanks [~t3rmin4t0r] for recommending this. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20468) Add ability to skip creating druid bitmap indexes for specific string dimensions

2018-08-27 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-20468:
---

 Summary: Add ability to skip creating druid bitmap indexes for 
specific string dimensions
 Key: HIVE-20468
 URL: https://issues.apache.org/jira/browse/HIVE-20468
 Project: Hive
  Issue Type: New Feature
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


Currently we create bitmap index for all druid dimensions. 
For some columns (e.g Free form text, high cardinality columns that are rarely 
filtered upon), It may be beneficial to skip creating druid bitmap index and 
save disk space.  

In druid https://github.com/apache/incubator-druid/pull/5402 added support for 
creating string dimension columns without bitmap indexes. 
This task is to add similar option when indexing data from hive. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207983
---


Ship it!




Ship It!

- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
>  68f07e2569b6531cf3e18919209aed1a17e88bf7 
>   
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207981
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Line 200 (original), 200 (patched)


Seems it can be static since it doesn't refer to any variables outside. 
Maybe I miss anything?


- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>

Review Request 68525: HIVE-20296 Improve HivePointLookupOptimizerRule to be able to extract from more sophisticated contexts

2018-08-27 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68525/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-20296
https://issues.apache.org/jira/browse/HIVE-20296


Repository: hive-git


Description
---

generalized rule to extract INs from more complex filter conditions as well.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
 eff9a312aaa09a889f0c6a045bd813d1fb633956 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/rules/TestHivePointLookupOptimizerRule.java
 PRE-CREATION 
  ql/src/test/results/clientpositive/druid/druidmini_test_ts.q.out 
a8e6894a9786318ed4362ee37d918a3699f074a0 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 
260ba1cbddee7f0946f0cdec1070359ab2a1d2aa 
  ql/src/test/results/clientpositive/perf/spark/query15.q.out 
67684f6b0bc44c0cae6107be94d131a083eca0e1 
  ql/src/test/results/clientpositive/perf/spark/query47.q.out 
690b1054c12f7d588015afc301802d9d2da2d0b9 
  ql/src/test/results/clientpositive/perf/spark/query57.q.out 
51e644a87bf4befbe4368cdf03b5eaab6d4f2049 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 
e1eca99d95e13070f901b75a406391435b2b4f1d 
  ql/src/test/results/clientpositive/perf/tez/query47.q.out 
d034ea9433a3b1c54c545edf526c215bf79388e1 
  ql/src/test/results/clientpositive/perf/tez/query57.q.out 
42cbbdc2a4d8bc469c9d91867faf21fc94a057ea 


Diff: https://reviews.apache.org/r/68525/diff/1/


Testing
---


Thanks,

Zoltan Haindrich

[jira] [Created] (HIVE-20467) Allow IF NOT EXISTS/IF EXISTS in Resource plan creation/drop

2018-08-27 Thread Miklos Gergely (JIRA) 
Miklos Gergely created HIVE-20467:
-

 Summary: Allow IF NOT EXISTS/IF EXISTS in Resource plan 
creation/drop
 Key: HIVE-20467
 URL: https://issues.apache.org/jira/browse/HIVE-20467
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.1
Reporter: Miklos Gergely
 Fix For: 3.0.1






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68523: Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience

2018-08-27 Thread denys kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68523/
---

Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-20466
https://issues.apache.org/jira/browse/HIVE-20466


Repository: hive-git


Description
---

When a create function statement is submitted, it may fail with the following 
error:

Error while processing statement: FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.FunctionTask

This is not a user-friendly error message.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 44591842bb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java 6f8a8f5504 
  ql/src/test/queries/clientnegative/create_unknown_permanent_udf.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/create_unknown_permanent_udf.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68523/diff/1/


Testing
---

added negative qtest to cover this scenario


Thanks,

denys kuzmenko

[jira] [Created] (HIVE-20466) Improve org.apache.hadoop.hive.ql.exec.FunctionTask Experience

2018-08-27 Thread denys kuzmenko (JIRA) 
denys kuzmenko created HIVE-20466:
-

 Summary: Improve org.apache.hadoop.hive.ql.exec.FunctionTask 
Experience
 Key: HIVE-20466
 URL: https://issues.apache.org/jira/browse/HIVE-20466
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: denys kuzmenko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 68518: ProxyFileSystem.listStatusIterator function override required once migrated to Hadoop 3.2.0+

2018-08-27 Thread denys kuzmenko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68518/
---

Review request for hive, Marta Kuczora and Peter Vary.


Bugs: HIVE-20465
https://issues.apache.org/jira/browse/HIVE-20465


Repository: hive-git


Description
---

Few list_bucket_query_oneskew_*.q tests started to fail. It is the side effect 
of this Hadoop change:
http://github.mtv.cloudera.com/CDH/hadoop/commit/073a38ee09f40b25677cc49eff777241c8fb2eba#diff-4bf68d76a459a69bbb0affbf579ebcf3

In Hive there is the ProxyFileSystem class which has the 'swizzleParamPath' 
method. This method will replace the 'pfile' prefix to 'file' in the path. The 
ProxyFileSystem will be used as file system during the test execution and this 
class implements some methods of the hadoop FileSystem class. Before the linked 
Hadoop patch, the file status are fetched with the 'listStatus' method call 
which is implemented in the ProxyFileSystem. But the patch changed this code 
path and now it calls the FileSystem.listStatusIterator method which is not 
implemented in the ProxyFileSystem, so it will fall back to the Hadoop 
implementation which will throw error for the 'pfile' prefix.


Diffs
-

  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
a82b2f0564 


Diff: https://reviews.apache.org/r/68518/diff/1/


Testing
---


Thanks,

denys kuzmenko

[jira] [Created] (HIVE-20465) ProxyFileSystem.listStatusIterator function override required once migrated to Hadoop 3.2.0 +

2018-08-27 Thread denys kuzmenko (JIRA) 
denys kuzmenko created HIVE-20465:
-

 Summary: ProxyFileSystem.listStatusIterator function override 
required once migrated to Hadoop 3.2.0 +
 Key: HIVE-20465
 URL: https://issues.apache.org/jira/browse/HIVE-20465
 Project: Hive
  Issue Type: Test
Reporter: denys kuzmenko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)