date:20171031

Re: Review Request 63427: HIVE-17396

2017-10-31 Thread Lefty Leverenz


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63427/#review189778
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3017-3018 (patched)


Please spell out the acronyms TS, DPP, and nDVs in the parameter 
description.

Also, the description isn't clear to me -- "to control if ..." what?  
Something about reduction, or feeding?  (Forgive my ignorance, I'm just looking 
at the grammatical structure of the sentence along with the parameter name.)


- Lefty Leverenz


On Oct. 31, 2017, 1:19 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63427/
> ---
> 
> (Updated Oct. 31, 2017, 1:19 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Dynamic Semijoin Reduction : markSemiJoinForDPP marks unwanted semijoin 
> branches
> 
> In method markSemiJoinForDPP (HIVE-17399), the nDVs comparison should not 
> have equality as there is a chance that the values are same on both sides and 
> the branch is still marked as good when it shouldn't be.
> Add a configurable factor to see how useful this is if nDVs on smaller side 
> are only slightly less than that on TS side.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6631a6e45d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java da30c3b642 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q 6cc0a7f7a9 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
> 1a1a4d9b2d 
> 
> 
> Diff: https://reviews.apache.org/r/63427/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-17955) Issue with the 'like' function in Hive and Spark SQL. The following command should not fail as per the documentation: select like("Vishal Jaiswal", "Jaiswal");

2017-10-31 Thread Vishal Jaiswal (JIRA)

Vishal Jaiswal created HIVE-17955:
-

 Summary: Issue with the 'like' function in Hive and Spark SQL. The 
following command should not fail as per the documentation: select like("Vishal 
Jaiswal", "Jaiswal");
 Key: HIVE-17955
 URL: https://issues.apache.org/jira/browse/HIVE-17955
 Project: Hive
  Issue Type: Bug
Reporter: Vishal Jaiswal


Command: describe function like;
Result: like(str, pattern) - Checks if str matches pattern

Command: select like("Vishal Jaiswal", "Jaiswal");
Result: Query fails in Hive, also checked via Spark SQL




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

2017-10-31 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63458/
---

Review request for hive, Sergey Shelukhin and Thejas Nair.


Bugs: HIVE-16917
https://issues.apache.org/jira/browse/HIVE-16917


Repository: hive-git


Description
---

HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
a3c853a5c5a2a3e72ae0141d0ad836ec95516280 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
9b2ae57ee498e9b712cc87b16ba223995519611c 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
6354c8c5450eaa8ab91dd59387f2eddf0eb84389 
  
service/src/test/org/apache/hive/service/cli/TestCLIServiceConnectionLimits.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63458/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-17954) Implement create, alter and drop pool API's.

2017-10-31 Thread Harish Jaiprakash (JIRA)

Harish Jaiprakash created HIVE-17954:


 Summary: Implement create, alter and drop pool API's.
 Key: HIVE-17954
 URL: https://issues.apache.org/jira/browse/HIVE-17954
 Project: Hive
  Issue Type: Sub-task
Reporter: Harish Jaiprakash


Implement pool management commands:

CREATE POOL `resource_plan`.`pool_path` WITH
  ALLOC_FRACTION `fraction`
  QUERY_PARALLELISM `parallelism`
  SCHEDULING_POLICY `policy`;

ALTER POOL `resource_plan`.`pool_path` SET
  PATH = `new_path`,
  ALLOC_FRACTION = `fraction`,
  QUERY_PARALLELISM = `parallelism`,
  SCHEDULING_POLICY = `policy`;

DROP POOL `resource_plan`.`pool_path`;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-10-31 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/
---

(Updated Nov. 1, 2017, 1:15 a.m.)


Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql 16190dc0a1 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 4c35380af4 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
  metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql 0ee0f758f1 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 86f3e5c4a9 
  metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 1affa7a119 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 2453bb92c6 
  metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql 0ee0f758f1 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 47bfc5ae08 
  metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql aa27a6486b 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 5c770e2659 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 591be49af1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
81d6b859a6 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 f1258bab17 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 9df1b82d62 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
6a2ff75c84 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 306559c722 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
f0658a665e 
  ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 5cdfc9de3e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 62801c5385 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 27a1bd8f25 
  standalone-metastore/src/main/resources/package.jdo 296e273ec8 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 60531457b7 


Diff: https://reviews.apache.org/r/63346/diff/6/

Changes: https://reviews.apache.org/r/63346/diff/5-6/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17953) Metrics should move to destination atomically

2017-10-31 Thread Alexander Kolbasov (JIRA)

Alexander Kolbasov created HIVE-17953:
-

 Summary: Metrics should move to destination atomically
 Key: HIVE-17953
 URL: https://issues.apache.org/jira/browse/HIVE-17953
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


HIVE-17563 reimplemented metrics using native nio interfaces. It used the 
assumption that{{Files.move()}} is atomic operation. It turns out that by 
default it isn't, unless {{ATOMIC_MOVE}} option is specified. Otherwise the 
destination file is unlinked and then the source file is copied.

This may cause test failure since the file may be temporarily unavailable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17952) Fix license headers to avoid dangling javadoc warnings

2017-10-31 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-17952:


 Summary: Fix license headers to avoid dangling javadoc warnings
 Key: HIVE-17952
 URL: https://issues.apache.org/jira/browse/HIVE-17952
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Priority: Trivial


All license headers starts with "/**" which are assumed to be javadocs and IDE 
warns about dangling javadoc pointing to license headers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17951) Clarify OrcSplit.hasBase() etc

2017-10-31 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-17951:
-

 Summary: Clarify OrcSplit.hasBase() etc
 Key: HIVE-17951
 URL: https://issues.apache.org/jira/browse/HIVE-17951
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


With HIVE-17089,  the meaning of

{code:java}
OrcSplit.hasBase()
OrcSplit.isOriginal()
OrcSplit.isAcid()

{code}
have shifted somewhat.
Need to clarify definitions/uses.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Hive metastore TLP

2017-10-31 Thread Padma Penumarthy

Hi,

We are exploring the possibility of using hive metastore as metastore for 
Apache Drill, to store and
manage our metadata. While investigating, I came across the metastore TLP 
proposal,
https://cwiki.apache.org/confluence/display/Hive/Metastore+TLP+Proposal.

I want to find out what is the timeline for this to be available to end users ?
when will it be production ready ?
If we integrate with hive and design/develop using current metastore client 
API, do we run
into risk of incompatibility in the future ? Our goal is to eventually move to 
using hive metastore only for the
reasons correctly identified in your wiki i.e. users do not have to install 
hive to be able to use Drill.

Any input/feedback you can provide to us will be very helpful for us to figure 
out our plans.

Thanks
Padma

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189762
---



- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
> 2c37a51cf4 
>   shims/scheduler/pom.xml 0eadb69435 
>   spark-client/pom.xml 784d908e4a 
>   standalone-metastore/pom.xml 8df622f179 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
>  c10e36f94a 
>   testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 
> 
> 
> Diff: https://reviews.apache.org/r/63394/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189747
---




hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
Lines 119-120 (patched)


You mean we want to support NULL for Hbase table locations? I think we 
allow this 

`LOCATION` varchar(4000) DEFAULT NULL.

Seems we just need to correct the comments.



itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
Lines 106 (patched)


I think this will be needed on the cluster as well so hbase will use the 
right native library. Hbase has a doc for that but I can't find it right now.


- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
> 2c37a51cf4 
>   shims/scheduler/pom.xml 0eadb69435 
>   spark-client/pom.xml 784d908e4a 
>   standalone-metastore/pom.xml 8df622f179 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
>  c10e36f94a 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

(Updated Oct. 31, 2017, 9:56 p.m.)


Review request for hive.


Changes
---

Address comments.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs (updated)
-

  common/pom.xml 868e14dbc3 
  hbase-handler/pom.xml 7f57b77168 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f0 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa9 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed6562 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c1 
  itests/hcatalog-unit/pom.xml bb6b10554f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f9 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514 
  itests/hive-minikdc/pom.xml 95d2614901 
  itests/hive-unit-hadoop2/pom.xml 339a194f2d 
  itests/hive-unit/pom.xml 14409832c8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b7 
  itests/qtest-accumulo/pom.xml 40d0a749c8 
  itests/qtest-spark/pom.xml a506f7f081 
  itests/qtest/pom.xml 02664f3792 
  itests/util/pom.xml e6dc09fb8f 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a89085 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925 
  llap-server/pom.xml 47a04cc310 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b6 
  metastore/pom.xml 04c6f47879 
  pom.xml 006e8f8611 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd 
  ql/src/test/results/clientnegative/external1.q.out 661d669206 
  ql/src/test/results/clientnegative/external2.q.out eb5518c017 
  serde/pom.xml 7419cfb17a 
  shims/0.23/pom.xml 3ff1d38776 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11d 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf4 
  shims/scheduler/pom.xml 0eadb69435 
  spark-client/pom.xml 784d908e4a 
  standalone-metastore/pom.xml 8df622f179 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a 
  testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 


Diff: https://reviews.apache.org/r/63394/diff/2/

Changes: https://reviews.apache.org/r/63394/diff/1-2/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board



> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/pom.xml
> > Line 82 (original), 82 (patched)
> > 
> >
> > scope=test?

Actually the code does have a reference to it. Not only for test.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 46 (patched)
> > 
> >
> > Might be a good idea to extend DefaultHiveMetaHook to take advantage of 
> > other extension points.

I'm not very faimilar with the implementation for insert operation. So I will 
delay that to later jira.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 57 (patched)
> > 
> >
> > Any reason to not do this in this patch?

Actually they are closed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
> > Line 742 (original), 742 (patched)
> > 
> >
> > Do we need to retain this for 0.23? That support is long gone :)

I will leave this cleanup for later jira since seems other places also have 
0.23 support.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
> > Lines 106 (patched)
> > 
> >
> > Is this test only or on cluster also we need to set up this property?

This is needed for the cluster as well so habase will pick up right native lib.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > pom.xml
> > Line 841 (original), 907 (patched)
> > 
> >
> > I don't think we need to depend on hbase-sever. If this is only for 
> > unit tests, shall we add test

Actually we need for the production code.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > serde/pom.xml
> > Line 155 (original), 161 (patched)
> > 
> >
> > Do we need this as well? Proabably not.

It's actually needed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > shims/0.23/pom.xml
> > Line 67 (original), 73 (patched)
> > 
> >
> > This shall not be needed.

This is needed for getMiniDfs() call.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/pom.xml
> > Line 179 (original), 179 (patched)
> > 
> >
> > Probably not needed.

It's actually needed.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189595
---


On Oct. 28, 2017, 11:54 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 28, 2017, 11:54 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc321be1e4008ba1575568bbc81032791 
>   hbase-handler/pom.xml 7f57b771686aeb09d4d58a4ae591110c10260519 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3abfd78bb4139eb471beeb0b66211f2a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b1a2654b8865d10645963cbe69eaa33 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f00d4a8ffe4a3649486d0dfe1e19ae68 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c258983337a63a27415de990d56aa6f 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913bf5ec0b1ee2cb7408c3da1d0785777 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa99ee751fc8a25cce2e1a0af67e015c4 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60524640cd546d15cd88bacc28ceb36b 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
>

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

2017-10-31 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63230/
---

(Updated Oct. 31, 2017, 9:13 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a5c5 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
 fdb660af03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AmPluginNode.java 35d380c92b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
d978a25b14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapPluginEndpointClientImpl.java
 45c3e38dcc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
a326db3ab0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SessionExpirationTracker.java 
da93a3a791 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java b67c933b19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
9b4714f1d7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
613522357e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
144816862d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
81d6b859a6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 00501eef93 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
35e5710629 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 209cf57a6a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
59efd43be6 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
258a865fef 


Diff: https://reviews.apache.org/r/63230/diff/6/

Changes: https://reviews.apache.org/r/63230/diff/5-6/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17950) Implement resource plan fetching from metastore

2017-10-31 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-17950:


 Summary: Implement resource plan fetching from metastore
 Key: HIVE-17950
 URL: https://issues.apache.org/jira/browse/HIVE-17950
 Project: Hive
  Issue Type: Sub-task
Reporter: Prasanth Jayachandran






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17949) itests compile is busted on branch-1.2

2017-10-31 Thread Mithun Radhakrishnan (JIRA)

Mithun Radhakrishnan created HIVE-17949:
---

 Summary: itests compile is busted on branch-1.2
 Key: HIVE-17949
 URL: https://issues.apache.org/jira/browse/HIVE-17949
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 1.2.3
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan


{{commit 18ddf46e0a8f092358725fc102235cbe6ba3e24d}} on {{branch-1.2}} was for 
{{Preparing for 1.2.3 development}}. This should have also included 
corresponding changes to all the pom-files under {{itests}}. As it stands now, 
the build fails with the following:

{noformat}
[ERROR]   location: class org.apache.hadoop.hive.metastore.api.Role
[ERROR] 
/Users/mithunr/workspace/dev/hive/apache/branch-1.2/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:[512,19]
 no suitable method found for 
updatePartitionStatsFast(org.apache.hadoop.hive.metastore.api.Partition,org.apache.hadoop.hive.metastore.Warehouse)
[ERROR] method 
org.apache.hadoop.hive.metastore.MetaStoreUtils.updatePartitionStatsFast(org.apache.hadoop.hive.metastore.api.Partition,org.apache.hadoop.hive.metastore.Warehouse,org.apache.hadoop.hive.metastore.api.EnvironmentContext)
 is not applicable
[ERROR]   (actual and formal argument lists differ in length)
[ERROR] method 
org.apache.hadoop.hive.metastore.MetaStoreUtils.updatePartitionStatsFast(org.apache.hadoop.hive.metastore.api.Partition,org.apache.hadoop.hive.metastore.Warehouse,boolean,org.apache.hadoop.hive.metastore.api.EnvironmentContext)
 is not applicable
[ERROR]   (actual and formal argument lists differ in length)
[ERROR] method 
org.apache.hadoop.hive.metastore.MetaStoreUtils.updatePartitionStatsFast(org.apache.hadoop.hive.metastore.api.Partition,org.apache.hadoop.hive.metastore.Warehouse,boolean,boolean,org.apache.hadoop.hive.metastore.api.EnvironmentContext)
 is not applicable
[ERROR]   (actual and formal argument lists differ in length)
[ERROR] method 
org.apache.hadoop.hive.metastore.MetaStoreUtils.updatePartitionStatsFast(org.apache.hadoop.hive.metastore.partition.spec.PartitionSpecProxy.PartitionIterator,org.apache.hadoop.hive.metastore.Warehouse,boolean,boolean,org.apache.hadoop.hive.metastore.api.EnvironmentContext)
 is not applicable
[ERROR]   (actual and formal argument lists differ in length)
[ERROR] 
/Users/mithunr/workspace/dev/hive/apache/branch-1.2/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreWithEnvironmentContext.java:[181,45]
 incompatible types: org.apache.hadoop.hive.metastore.api.EnvironmentContext 
cannot be converted to boolean
[ERROR] 
/Users/mithunr/workspace/dev/hive/apache/branch-1.2/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreWithEnvironmentContext.java:[190,45]
 incompatible types: org.apache.hadoop.hive.metastore.api.EnvironmentContext 
cannot be converted to boolean
[ERROR] 
/Users/mithunr/workspace/dev/hive/apache/branch-1.2/itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java:[53,26]
 cannot find symbol
[ERROR]   symbol:   class MiniZooKeeperCluster
[ERROR]   location: class org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hive-it-unit
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[CVE-2017-12625] Apache Hive information disclosure vulnerability for column masking

2017-10-31 Thread Jesus Camacho Rodriguez

CVE-2017-12625: Apache Hive information disclosure vulnerability for column 
masking

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected: Hive 2.1.0 to 2.3.0

Description:
Hive exposes an interface through which masking policies can be defined on 
tables or
views, e.g., using Apache Ranger. When a view is created over a given table, the
policy enforcement does not happen correctly on the table for masked columns.

Mitigation:
2.3.0 users should upgrade to 2.3.1
2.2.0 users should upgrade to 2.3.1, obtain the latest source from git for 
branch-2.2
or apply this patch which will be included from 2.2.1
https://git1-us-west.apache.org/repos/asf?p=hive.git;a=commit;h=0e795debddf261b0ac6ace90e2d774f9a99b7f4b
2.1.x users should upgrade to 2.3.1, obtain the latest source from git for 
branch-2.1
or apply this patch which will be included from 2.1.2
https://git1-us-west.apache.org/repos/asf?p=hive.git;a=commit;h=6db9fd6e43f6eef3c9d1ca8e324b2edaa54fb0d3

To mitigate this vulnerability until Hive is upgraded to a new version, there 
are two
possible options. These steps need to be done manually in Ranger / Hive.
1) Restrict users from creating views on tables with column masking rules 
defined. For
this in Ranger Hive Policy:
 - Users should not have SELECT permission for those Table columns with masking 
rules
defined.
 - Give SELECT permission only for those columns without masking rules defined.
2) Review the Hive Column Masking Policies maintained in Ranger for the tables. 
Then
check in Hive if views that read those tables have been defined.
If present, either change the view definition so those columns are not selected 
or
directly drop those views.

Credit:
This issue was reported by Suja Santhosh of Hortonworks.


If you have any question, please reach out to us in the Hive dev list.

Regards,

The Apache Hive Team

Re: Review Request 63381: HIVE-17926: Support triggers for non-pool sessions

2017-10-31 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63381/
---

(Updated Oct. 31, 2017, 7:27 p.m.)


Review request for hive and Sergey Shelukhin.


Changes
---

Avoids running all tests for TestTriggersNoTezSessionPool


Bugs: HIVE-17926
https://issues.apache.org/jira/browse/HIVE-17926


Repository: hive-git


Description
---

HIVE-17926: Support triggers for non-pool sessions


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractJdbcTriggersTest.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersNoTezSessionPool.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersTezSessionPoolManager.java
 aeca663 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java b67c933 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
9b4714f 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 2c4fe7f 


Diff: https://reviews.apache.org/r/63381/diff/4/

Changes: https://reviews.apache.org/r/63381/diff/3-4/


Testing
---


Thanks,

Prasanth_J

Re: Client compatibility for MetaStoreFilterHook

2017-10-31 Thread Thejas Nair

Hi Alex,
Sorry about the delay in responding. I have replied in the jira.

Thanks,
Thejas



On Mon, Oct 30, 2017 at 5:41 PM, Alexander Kolbasov 
wrote:

> Looking at https://issues.apache.org/jira/browse/HIVE-9350 (Add ability
> for
> HiveAuthorizer implementations to filter out results of 'show tables',
> 'show databases') I see that it changed methods of MetaStoreFilterHook to
> throw exceptions which id didn't throw before. This means that any clients
> using it require recompilation (and potential code changes to handle
> exceptions) which is a compatibility issue. This was done in 1.2.0 - IN am
> wondering how such compatibility issues were handled in Hive 1 and Hive 2
> branches?
>
> Thanks for clarification,
>
> - Alex
>

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

2017-10-31 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63230/
---

(Updated Oct. 31, 2017, 6:54 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a3c853a5c5 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
 fdb660af03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AmPluginNode.java 35d380c92b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
d978a25b14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapPluginEndpointClientImpl.java
 45c3e38dcc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
a326db3ab0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SessionExpirationTracker.java 
da93a3a791 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java b67c933b19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
9b4714f1d7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
613522357e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
144816862d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
81d6b859a6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 00501eef93 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
35e5710629 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 209cf57a6a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
59efd43be6 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
258a865fef 


Diff: https://reviews.apache.org/r/63230/diff/5/

Changes: https://reviews.apache.org/r/63230/diff/4-5/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17948) Hive 2.3.2 Release Planning

2017-10-31 Thread Sahil Takiar (JIRA)

Sahil Takiar created HIVE-17948:
---

 Summary: Hive 2.3.2 Release Planning
 Key: HIVE-17948
 URL: https://issues.apache.org/jira/browse/HIVE-17948
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: Sahil Takiar
Assignee: Sahil Takiar
 Fix For: 2.3.2


Release planning for Hive 2.3.2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[DISCUSS] Hive 2.3.2 bugfix release

2017-10-31 Thread Sahil Takiar

 Hello,

There is a bugfix (HIVE-17831) that is required by other projects to get
correct information from Hive hooks, but it did not land in the 2.3.1
release due to time constraints. I am thinking on doing a quick bugfix
release to unblock Sentry on its next major release.

If there are no objections, I'd like to cut a RC in a couple of days. If
you'd like to land in any critical bug fixes, please feel free to do so
(subject to the usual restrictions that apply to a bug fix release: no
major changes/db changes/breaking changes). I will also be looking for
critical bug fixes from branch-2 and master and will try to get them into
the 2.3.2 release.

I'll wait for a couple of days for any response here, and start a vote
thread.

Thanks,
Sahil Takiar

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

2017-10-31 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62280/
---

(Updated Oct. 31, 2017, 3:43 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

address review comments; relatively high volume of q.out changes because the 
zeros are coming back...


Bugs: HIVE-16827
https://issues.apache.org/jira/browse/HIVE-16827


Repository: hive-git


Description
---

this was originally part of HIVE-13567


Diffs (updated)
-

  
accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
 59bca5024d 
  common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
a73893faff 
  contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
  contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
  contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
  contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
  contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
  data/scripts/q_test_init_src.sql 56b44e0a5d 
  
hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out 
e8927e9d64 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
 e55b1c257e 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
f50f4af817 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
 660cebba5f 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 ba0e83d562 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 2ababb1eec 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
ad2baa2e26 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
2edf749399 
  itests/src/test/resources/testconfiguration.properties 4b521f052b 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
70d33ff93e 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
7337c1ca6c 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
ef097acb0f 
  pom.xml 006e8f8611 
  ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 583d3d3893 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 1f286887e4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 2331498781 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java c333c494a1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 682b42c202 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java ab495cfae8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 36a5eff1e3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java cd2d091a23 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 70656feea7 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
78e83af4f0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
768640c544 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
dbf4b8da01 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/QueryPlanPostProcessor.java 
91c6c0050b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
3a20cfe7ac 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
dc433fed22 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SkewJoinResolver.java 
2f9783ed18 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
0f7ef8b46d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
6a2ff75c84 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
7a0d4a752e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 01cb2b31a5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
5f2a34ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
9309fbd0e3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e40ec5d909 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java a63f709eef 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
6f21cae8d6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 97f323f4b7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java 842fd1a411 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 550e6f867f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/IStatsGatherDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 30d99123fc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java 77c04f6c6e

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

2017-10-31 Thread Zoltan Haindrich



- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62280/#review189221
---


On Oct. 3, 2017, 8:06 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62280/
> ---
> 
> (Updated Oct. 3, 2017, 8:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16827
> https://issues.apache.org/jira/browse/HIVE-16827
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> this was originally part of HIVE-13567
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024e4d6f21e3b1d24a5efcc1325e64d6e 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> b7dc88c93984a37a5df7ec8258c3e1e375cf7878 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 
> 6876ca8775098175155111c25d5dba4db63b3b1b 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 
> 79cf8fe1e5d159a9fa013dfd616a84918709a0f9 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out 
> fec58ef026074b2b2d0c9a4f91cc219e4e21421c 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 
> 1131478a7b6694a106d41206042fe6dee99eb8a2 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 
> 8d3b95ece81e55193d92cbc39960bf378990e256 
>   data/scripts/q_test_init_src.sql 56b44e0a5df4d32a6a9318a760ce2debdad00d50 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  e8927e9d64a7978355b387100e4e30608c34eb7b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257ef548000d96a2d7a1a14c39dc34bc0b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> f50f4af817f000f6cc59133d5966899e79d67c3b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f8a28abcd6b10b62d5982468a92b962 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  ba0e83d5623f6cd9d7ada998d47826883cb2aca4 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec51f49a4a92367deaa87ee7b2d12797 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e265d2d0ffb94859e5141e4b38f2909b5 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 2edf7493990572ab3a9d6f1a7c4407de91bef39c 
>   itests/src/test/resources/testconfiguration.properties 
> 038487f134f2c24d0b957404de6afb1766a2cb4a 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 65551ad169d8151fce297d8a44fd30168a682ad0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> f4af6f497927d61517e263521ef97a27c616e14c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> a147a2590d6de1fe161c7b02f043f179243cf83c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b5e4bf086e9b2659ebb3b221269ee4a69a0465ae 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/merge/ColumnStatsMergerFactory.java
>  66be52413956b261373b2132c5678a204662c79e 
>   pom.xml 52e53012b9ca91d953baa257e1dd74ba86220455 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 
> 583d3d3893298e823e27e82a13b252f799bbef79 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 
> e3b0d1eaea337d318918c9e9ea07be53bffee189 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> bc265eb790ab8f9517ecfdd56b0634e47092ca35 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
> 9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 
> 4db68066d6dbc557180a2fefe63069a924fede65 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
> 6193b900e085bbe013f0526ee8c351cdff4e655a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 
> e9c69d94d5b41748cde08eaf2636c513c5731929 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> ae637279998e7e4983d6c9abf8c0af75b92d30da 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 436a2fe73bc91f59ea83490291fa3668a6442930 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  61f6a7c4ff38447db0ac2610e7308f4e710580ab 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b87454bf37b1a4c68327407cada6b37232 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 25cca7b4be124f6b6a6ea645ef887f6cf5cfa8d6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/QueryPlanPostProcessor.java

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

2017-10-31 Thread Zoltan Haindrich



> On Oct. 10, 2017, 11:49 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java
> > Lines 160 (patched)
> > 
> >
> > we want to just flip the state flag and keep whatever stats we have.
> 
> Zoltan Haindrich wrote:
> I would like to remove stats which might be misleading; outputs which 
> contain a value for `numRows` ; but basic_stats are false could cause a 
> headache for the explains readers...
> 
> It turned out that removing invalid stats has a positive sideeffect: 
> until distinct_windowing_no_cbo.q estimated 1 row for a table containing 10k 
> rows; now it changed to 84k which seems better to me.
> 
> Anyway...there is an issue with the `LOAD` statements; seems like 
> `hive.stats.autogather` doesn't work on it properly (this will be a followup)
> 
> Ashutosh Chauhan wrote:
> Having incorrect but approximate stats are much more useful for optimizer 
> than having no stats at all. We keep state so that we don't use it to answer 
> queries but approximate stats have role to play.
> 
> Load statement is suppose to set state to false. Since, it doesn't scan 
> data, stats can be computed for this statement.

but in case we want to keep "incorrect" stats...why don't update them? - they 
might be just partially correct; but in case its possible the stat system 
should update them or not?


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62280/#review187592
---


On Oct. 3, 2017, 8:06 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62280/
> ---
> 
> (Updated Oct. 3, 2017, 8:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16827
> https://issues.apache.org/jira/browse/HIVE-16827
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> this was originally part of HIVE-13567
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024e4d6f21e3b1d24a5efcc1325e64d6e 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> b7dc88c93984a37a5df7ec8258c3e1e375cf7878 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 
> 6876ca8775098175155111c25d5dba4db63b3b1b 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 
> 79cf8fe1e5d159a9fa013dfd616a84918709a0f9 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out 
> fec58ef026074b2b2d0c9a4f91cc219e4e21421c 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 
> 1131478a7b6694a106d41206042fe6dee99eb8a2 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 
> 8d3b95ece81e55193d92cbc39960bf378990e256 
>   data/scripts/q_test_init_src.sql 56b44e0a5df4d32a6a9318a760ce2debdad00d50 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  e8927e9d64a7978355b387100e4e30608c34eb7b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257ef548000d96a2d7a1a14c39dc34bc0b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> f50f4af817f000f6cc59133d5966899e79d67c3b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f8a28abcd6b10b62d5982468a92b962 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  ba0e83d5623f6cd9d7ada998d47826883cb2aca4 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec51f49a4a92367deaa87ee7b2d12797 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e265d2d0ffb94859e5141e4b38f2909b5 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 2edf7493990572ab3a9d6f1a7c4407de91bef39c 
>   itests/src/test/resources/testconfiguration.properties 
> 038487f134f2c24d0b957404de6afb1766a2cb4a 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 65551ad169d8151fce297d8a44fd30168a682ad0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> f4af6f497927d61517e263521ef97a27c616e14c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> a147a2590d6de1fe161c7b02f043f179243cf83c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b5e4bf086e9b2659ebb3b221269ee4a69a0465ae 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/merge/ColumnStatsMergerFactory.java
>  66be52413956b261373b2132c5678a204662c79e 
>

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

2017-10-31 Thread Zoltan Haindrich



> On Oct. 25, 2017, 11:10 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
> > Lines 127 (patched)
> > 
> >
> > Why do we need location? Stats tmp dir is in desc already.

this is a problematic thing:

* in case of CTAS the table object alike thing which is accessible during task 
generation time didn't have location filled out.
* DDLTask fills it out
* table location is needed for footerscan to work


> On Oct. 25, 2017, 11:10 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> > Lines 1486 (patched)
> > 
> >
> > I dont see this table obj being used here. Shall we remove this, 
> > especially since its making a metastore call.

I deliberately moved it to here; the table ref is used to construct StatsWork
I think its better to make the metastore calls before starting the plan 
execution.
These metastore calls pre-existed; IIRC this new way avoids 1 metastore call


> On Oct. 25, 2017, 11:10 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out
> > Line 295 (original)
> > 
> >
> > For create table we set basic stats as 0 and state as true, seems to be 
> > not happening here.

temporary tables are kinda problematic...I wanted to address it later; now that 
I've turned this back on...they have numRows 0 entries - which is incorrect
I wanted to fully disable this thing - to prevent issues arising from the fact 
that it misbehaves.


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62280/#review189221
---


On Oct. 3, 2017, 8:06 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62280/
> ---
> 
> (Updated Oct. 3, 2017, 8:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16827
> https://issues.apache.org/jira/browse/HIVE-16827
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> this was originally part of HIVE-13567
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024e4d6f21e3b1d24a5efcc1325e64d6e 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> b7dc88c93984a37a5df7ec8258c3e1e375cf7878 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 
> 6876ca8775098175155111c25d5dba4db63b3b1b 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 
> 79cf8fe1e5d159a9fa013dfd616a84918709a0f9 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out 
> fec58ef026074b2b2d0c9a4f91cc219e4e21421c 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 
> 1131478a7b6694a106d41206042fe6dee99eb8a2 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 
> 8d3b95ece81e55193d92cbc39960bf378990e256 
>   data/scripts/q_test_init_src.sql 56b44e0a5df4d32a6a9318a760ce2debdad00d50 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  e8927e9d64a7978355b387100e4e30608c34eb7b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257ef548000d96a2d7a1a14c39dc34bc0b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> f50f4af817f000f6cc59133d5966899e79d67c3b 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f8a28abcd6b10b62d5982468a92b962 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  ba0e83d5623f6cd9d7ada998d47826883cb2aca4 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec51f49a4a92367deaa87ee7b2d12797 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e265d2d0ffb94859e5141e4b38f2909b5 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 2edf7493990572ab3a9d6f1a7c4407de91bef39c 
>   itests/src/test/resources/testconfiguration.properties 
> 038487f134f2c24d0b957404de6afb1766a2cb4a 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 65551ad169d8151fce297d8a44fd30168a682ad0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> f4af6f497927d61517e263521ef97a27c616e14c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> a147a2590d6de1fe161c7b02f043f179243cf83c 
>

Review Request 63442: HIVE-17934 Merging Statistics are promoted to COMPLETE (most of the time)

2017-10-31 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63442/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17934
https://issues.apache.org/jira/browse/HIVE-17934


Repository: hive-git


Description
---

* remove the reactive stat state guessing method
* make the guessing only work when a new object is created
* change the way stat objects are merged

this patch will most probably break almost all qtest outputs


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 7a3fae65e8c1c0090b353cdd674f9b67164695e0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 a4f60accce852d13a08e21261ac3a8d8ad7121e2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 
8ffb4ce44b71e949577bc0bb7d17aa3ee9bed92f 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 
9ffd12ad3c248176024b573f2d3eb23cab06ee05 
  ql/src/test/queries/clientpositive/stats_empty_partition2.q PRE-CREATION 
  ql/src/test/results/clientpositive/stats_empty_partition2.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/63442/diff/1/


Testing
---


Thanks,

Zoltan Haindrich

[GitHub] hive pull request #269: fix small bug: when disable partition statistics in ...

2017-10-31 Thread hejian991

GitHub user hejian991 opened a pull request:

https://github.com/apache/hive/pull/269

fix small bug: when disable partition statistics in hive-site.xml, buâ¦

â¦t it does not work.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hejian991/hive 
fixbug_MetaStoreUtils_requireCalStats

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #269


commit 2334adc2ad687e516ac9409dce1e3e0ebe5e6113
Author: hejian991 <807334...@qq.com>
Date:   2017-10-31T13:23:49Z

fix small bug: when disable partition statistics in hive-site.xml, but it 
does not work.




---

[jira] [Created] (HIVE-17947) Concurrent inserts might fail for ACID table since HIVE-17526 on branch-1

2017-10-31 Thread Daniel Voros (JIRA)

Daniel Voros created HIVE-17947:
---

 Summary: Concurrent inserts might fail for ACID table since 
HIVE-17526 on branch-1
 Key: HIVE-17947
 URL: https://issues.apache.org/jira/browse/HIVE-17947
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0
Reporter: Daniel Voros
Assignee: Daniel Voros
Priority: Blocker


HIVE-17526 (only on branch-1) disabled conversion to ACID if there are *_copy_N 
files under the table, but the filesystem checks introduced there are running 
for every insert since the MoveTask in the end of the insert will call 
alterTable eventually.

The filename checking also recurses into staging directories created by other 
inserts. If those are removed while listing the files, it leads to the 
following exception and failing insert:

{code}
java.io.FileNotFoundException: File 
hdfs://mycluster/apps/hive/warehouse/dvoros.db/concurrent_insert/.hive-staging_hive_2017-10-30_13-23-35_056_2844419018556002410-2/-ext-10001
 does not exist.
at 
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1081)
 ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1059)
 ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1004)
 ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1000)
 ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 ~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1018)
 ~[hadoop-hdfs-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1735) 
~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.fs.FileSystem$6.handleFileStat(FileSystem.java:1864) 
~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at org.apache.hadoop.fs.FileSystem$6.hasNext(FileSystem.java:1841) 
~[hadoop-common-2.7.3.2.6.3.0-235.jar:?]
at 
org.apache.hadoop.hive.metastore.TransactionalValidationListener.containsCopyNFiles(TransactionalValidationListener.java:226)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleAlterTableTransactionalProp(TransactionalValidationListener.java:104)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:63)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:55)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2478)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4145)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:4117)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at com.sun.proxy.$Proxy32.alter_table_with_environment_context(Unknown 
Source) [?:?]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:299)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table_with_environmentContext(SessionHiveMetaStoreClient.java:325)
 [hive-exec-2.1.0.2.6.3.0-235.jar:2.1.0.2.6.3.0-235]
at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at

Re: Review Request 63427: HIVE-17396

[jira] [Created] (HIVE-17955) Issue with the 'like' function in Hive and Spark SQL. The following command should not fail as per the documentation: select like("Vishal Jaiswal", "Jaiswal");

Review Request 63458: HIVE-16917: HiveServer2 guard rails - Limit concurrent connections from user

[jira] [Created] (HIVE-17954) Implement create, alter and drop pool API's.

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

[jira] [Created] (HIVE-17953) Metrics should move to destination atomically

[jira] [Created] (HIVE-17952) Fix license headers to avoid dangling javadoc warnings

[jira] [Created] (HIVE-17951) Clarify OrcSplit.hasBase() etc

Hive metastore TLP

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

[jira] [Created] (HIVE-17950) Implement resource plan fetching from metastore

[jira] [Created] (HIVE-17949) itests compile is busted on branch-1.2

[CVE-2017-12625] Apache Hive information disclosure vulnerability for column masking

Re: Review Request 63381: HIVE-17926: Support triggers for non-pool sessions

Re: Client compatibility for MetaStoreFilterHook

Re: Review Request 63230: HIVE-17841 implement applying the resource plan

[jira] [Created] (HIVE-17948) Hive 2.3.2 Release Planning

[DISCUSS] Hive 2.3.2 bugfix release

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

Re: Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

Review Request 63442: HIVE-17934 Merging Statistics are promoted to COMPLETE (most of the time)

[GitHub] hive pull request #269: fix small bug: when disable partition statistics in ...

[jira] [Created] (HIVE-17947) Concurrent inserts might fail for ACID table since HIVE-17526 on branch-1

29 matches

Site Navigation

Mail list logo

Footer information