[jira] [Created] (HIVE-18006) Optimize memory footprint of HLLDenseRegister

2017-11-07 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-18006:


 Summary: Optimize memory footprint of HLLDenseRegister
 Key: HIVE-18006
 URL: https://issues.apache.org/jira/browse/HIVE-18006
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


{code}
private double[] invPow2Register;
{code}
seems to add up memory when caching column stats (#table * #partition * #cols). 
This register can be pre-computed and stored as constant. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18010) Update hbase version

2017-11-07 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-18010:
---

 Summary: Update hbase version
 Key: HIVE-18010
 URL: https://issues.apache.org/jira/browse/HIVE-18010
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18011) hive lib too much repetition

2017-11-07 Thread zhaixiaobin (JIRA) 
zhaixiaobin created HIVE-18011:
--

 Summary: hive lib too much repetition
 Key: HIVE-18011
 URL: https://issues.apache.org/jira/browse/HIVE-18011
 Project: Hive
  Issue Type: Task
  Components: distribution
Affects Versions: 2.3.0
 Environment: Any Environment
Reporter: zhaixiaobin
 Fix For: 2.3.0


*Following is the lib directory of the hive, too many duplicate jar :*

-rw-r--r-- 1 root root  4368200 Dec  9  2015 accumulo-core-1.6.0.jar
-rw-r--r-- 1 root root   102069 Dec  9  2015 accumulo-fate-1.6.0.jar
-rw-r--r-- 1 root root57420 Dec  9  2015 accumulo-start-1.6.0.jar
-rw-r--r-- 1 root root   117409 Dec  9  2015 accumulo-trace-1.6.0.jar
-rw-r--r-- 1 root root62983 Dec  9  2015 activation-1.1.jar
-rw-r--r-- 1 root root   133957 Dec  9  2015 aether-api-0.9.0.M2.jar
-rw-r--r-- 1 root root26285 Dec 15  2016 aether-connector-file-0.9.0.M2.jar
-rw-r--r-- 1 root root52012 Dec 15  2016 aether-connector-okhttp-0.0.9.jar
-rw-r--r-- 1 root root   144866 Dec  9  2015 aether-impl-0.9.0.M2.jar
-rw-r--r-- 1 root root17703 Dec  9  2015 aether-spi-0.9.0.M2.jar
-rw-r--r-- 1 root root   133588 Dec  9  2015 aether-util-0.9.0.M2.jar
-rw-r--r-- 1 root root88458 Feb  3  2017 aircompressor-0.3.jar
-rw-r--r-- 1 root root85912 Sep  8  2016 airline-0.7.jar
-rw-r--r-- 1 root root  1034049 Dec  9  2015 ant-1.6.5.jar
-rw-r--r-- 1 root root  1997485 Dec  9  2015 ant-1.9.1.jar
{color}-rw-r--r-- 1 root root18336 Dec  9  2015 ant-launcher-1.9.1.jar
-rw-r--r-- 1 root root   374032 Dec  9  2015 antlr4-runtime-4.5.jar
-rw-r--r-- 1 root root   167761 Dec  9  2015 antlr-runtime-3.5.2.jar
-rw-r--r-- 1 root root31827 Aug 30  2016 apache-curator-2.7.1.pom
-rw-r--r-- 1 root root43033 Dec  9  2015 asm-3.1.jar
-rw-r--r-- 1 root root32693 Dec  9  2015 asm-commons-3.1.jar
-rw-r--r-- 1 root root21879 Dec  9  2015 asm-tree-3.1.jar
-rw-r--r-- 1 root root  5222951 Oct 18  2016 avatica-1.8.0.jar
-rw-r--r-- 1 root root20102 Oct 18  2016 avatica-metrics-1.8.0.jar
-rw-r--r-- 1 root root   436303 Dec  9  2015 avro-1.7.7.jar
-rw-r--r-- 1 root root   110600 Dec  9  2015 bonecp-0.8.0.RELEASE.jar
-rw-r--r-- 1 root root74175 Dec 15  2016 bytebuffer-collections-0.2.5.jar
-rw-r--r-- 1 root root  4085527 Oct 18  2016 calcite-core-1.10.0.jar
-rw-r--r-- 1 root root96585 Oct 18  2016 calcite-druid-1.10.0.jar
-rw-r--r-- 1 root root   481884 Oct 18  2016 calcite-linq4j-1.10.0.jar
-rw-r--r-- 1 root root60282 Sep  8  2016 classmate-1.0.0.jar
-rw-r--r-- 1 root root41123 Dec  9  2015 commons-cli-1.2.jar
-rw-r--r-- 1 root root58160 Dec  9  2015 commons-codec-1.4.jar
-rw-r--r-- 1 root root   588337 Dec  9  2015 commons-collections-3.2.2.jar
-rw-r--r-- 1 root root30595 Dec  9  2015 commons-compiler-2.7.6.jar
-rw-r--r-- 1 root root   378217 Dec  9  2015 commons-compress-1.9.jar
-rw-r--r-- 1 root root   160519 Dec  9  2015 commons-dbcp-1.4.jar
-rw-r--r-- 1 root root   167962 Sep  8  2016 commons-dbcp2-2.0.1.jar
-rw-r--r-- 1 root root   112341 Dec  9  2015 commons-el-1.0.jar
-rw-r--r-- 1 root root   279781 Dec  9  2015 commons-httpclient-3.0.1.jar
-rw-r--r-- 1 root root   185140 Dec  9  2015 commons-io-2.4.jar
-rw-r--r-- 1 root root   284220 Dec  9  2015 commons-lang-2.6.jar
-rw-r--r-- 1 root root   315805 Dec  9  2015 commons-lang3-3.1.jar
-rw-r--r-- 1 root root61829 Dec  9  2015 commons-logging-1.2.jar
-rw-r--r-- 1 root root   988514 Dec  9  2015 commons-math-2.2.jar
-rw-r--r-- 1 root root  2213560 Dec 15  2016 commons-math3-3.6.1.jar
-rw-r--r-- 1 root root96221 Dec  9  2015 commons-pool-1.5.4.jar
-rw-r--r-- 1 root root   108036 Sep  8  2016 commons-pool2-2.2.jar
-rw-r--r-- 1 root root   415578 Dec  9  2015 commons-vfs2-2.0.jar
-rw-r--r-- 1 root root79845 Dec  9  2015 compress-lzf-1.0.3.jar
-rw-r--r-- 1 root root   425111 Sep  8  2016 config-magic-0.9.jar
-rw-r--r-- 1 root root69500 Aug 30  2016 curator-client-2.7.1.jar
-rw-r--r-- 1 root root   186273 Aug 30  2016 curator-framework-2.7.1.jar
-rw-r--r-- 1 root root   270342 Aug 30  2016 curator-recipes-2.7.1.jar
-rw-r--r-- 1 root root59602 Dec 15  2016 curator-x-discovery-2.11.0.jar
-rw-r--r-- 1 root root   366748 Mar  7  2017 datanucleus-api-jdo-4.2.4.jar
-rw-r--r-- 1 root root  2016766 Mar  7  2017 datanucleus-core-4.1.17.jar
-rw-r--r-- 1 root root  1908681 Mar  7  2017 datanucleus-rdbms-4.1.19.jar
-rw-r--r-- 1 root root  2838580 Dec  9  2015 derby-10.10.2.0.jar
-rw-r--r-- 1 root root   583719 Dec 15  2016 derbyclient-10.11.1.1.jar
-rw-r--r-- 1 root root   266316 Dec 21  2015 derbynet-10.11.1.1.jar
-rw-r--r-- 1 root root79576 Dec  9  2015 disruptor-3.3.0.jar
-rw-r--r-- 1 root root15935 Aug 10  2016 
dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
-rw-r--r-- 1 root root   111673 Dec 15  2016 druid-api-0.9.2.jar
-rw-r--r-- 1 root root   197117 Dec 15  2016 druid-common-0.9.2.jar

[jira] [Created] (HIVE-18005) Improve size estimation for array() to be not 0

2017-11-07 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-18005:
---

 Summary: Improve size estimation for array() to be not 0
 Key: HIVE-18005
 URL: https://issues.apache.org/jira/browse/HIVE-18005
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Zoltan Haindrich


happens only in case the array is not from a column; and the array contains no 
column references

{code}
EXPLAIN
SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t

...
 Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE Column 
stats: COMPLETE
 ListSink
{code}






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18009) Multiple lateral view query is slow on hive on spark

2017-11-07 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-18009:
---

 Summary: Multiple lateral view query is slow on hive on spark
 Key: HIVE-18009
 URL: https://issues.apache.org/jira/browse/HIVE-18009
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


When running the query with multiple lateral view, HoS is busy with the 
compilation. GenSparkUtils has an efficient implementation of getChildOperator 
when we have diamond hierarchy in operator trees (lateral view in this case) 
since the node may be visited multiple times.

{noformat}
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:442)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at

Re: Review Request 63533: HIVE-17906 use kill query mechanics to kill queries in WM

2017-11-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63533/#review190417
---




ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 459 (patched)


needs to continue


- Sergey Shelukhin


On Nov. 8, 2017, 2:29 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63533/
> ---
> 
> (Updated Nov. 8, 2017, 2:29 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 
> fa0eedb1e2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 0dd1433a6c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 169991c639 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 0347e914bc 
> 
> 
> Diff: https://reviews.apache.org/r/63533/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 63533: HIVE-17906 use kill query mechanics to kill queries in WM

2017-11-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63533/
---

(Updated Nov. 8, 2017, 2:29 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java fa0eedb1e2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmTezSession.java 0dd1433a6c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
169991c639 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
0347e914bc 


Diff: https://reviews.apache.org/r/63533/diff/2/

Changes: https://reviews.apache.org/r/63533/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63586: Fix HIVE-17942: HiveAlterHandler should use the conf from HMS Handler

2017-11-07 Thread Janaki Lahorani via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63586/
---

(Updated Nov. 7, 2017, 9:19 p.m.)


Review request for hive, Alexander Kolbasov, Andrew Sherman, Sahil Takiar, and 
Vihang Karajgaonkar.


Repository: hive-git


Description
---

HMS handler thread local will have the configuration changes from the user 
local only to that connection.  HiveAlterHandler should use the thread local to 
pick up user's configuration changes.


Diffs
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreAlterColumnPar.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ccadac1ada6aaae884ab39f5d99e91b8c542404e 


Diff: https://reviews.apache.org/r/63586/diff/3/


Testing
---


Thanks,

Janaki Lahorani

[jira] [Created] (HIVE-18007) Address maven warnings

2017-11-07 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-18007:
---

 Summary: Address maven warnings
 Key: HIVE-18007
 URL: https://issues.apache.org/jira/browse/HIVE-18007
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 3.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


{code}
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hive:hive-metastore:jar:3.0.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found 
duplicate declaration of plugin org.apache.maven.plugins:maven-jar-plugin @ 
line 299, column 15
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hive:hive-standalone-metastore:jar:3.0.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.antlr:antlr3-maven-plugin is 
missing. @ line 538, column 15
[WARNING] It is highly recommended to fix these problems because they threaten 
the stability of your build.
[WARNING] For this reason, future Maven versions might no longer support 
building such malformed projects.
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18008) Add optimization rule to remove gby from right side of left semi-join

2017-11-07 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-18008:
--

 Summary: Add optimization rule to remove gby from right side of 
left semi-join
 Key: HIVE-18008
 URL: https://issues.apache.org/jira/browse/HIVE-18008
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Vineet Garg
Assignee: Vineet Garg


Group by (on same keys as semi join) as right side of Left semi join is 
unnecessary and could be removed. We see this pattern in subqueries with 
explicit distinct keyword e.g.

{code:sql}
explain select * from src b where b.key in (select distinct key from src a 
where a.value = b.value)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

What happened with HMS on Hbase project?

2017-11-07 Thread Alexander Kolbasov 
I am wondering what happened with HMS on HBase project? I saw a bunch of
slides telling about the progress and good POC results, but then looks like
it was abandoned for some reason. Were there any serious issues discovered
or benefits turned out to be insignificant?

- Alex

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/
---

(Updated Nov. 7, 2017, 8:55 p.m.)


Review request for hive, Harish Jaiprakash and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a26ea2115f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 996c0052cf 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
 012361a186 
  metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql f9b0765916 
  metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 054978e3e4 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
  metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql a13d976e37 
  metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1ceb7239eb 
  metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 03ee4cfd64 
  metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 06648540f0 
  metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql a13d976e37 
  metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql a94985a63e 
  metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql e80e6124fc 
  metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql be1bb1eee0 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
f29a20d44b 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 4df7c97795 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 a41e5a083c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 826ae565c1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
9954c249d4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
169991c639 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 f1258bab17 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 9df1b82d62 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
3415a23dec 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 8708f2a0b4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
04c9e4413d 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
0347e914bc 
  ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 2f314a6a8d 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 300ba7281d 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 7334a0c9fa 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 4fe70a35e1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 43d72e4cc9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f8ebc124c4 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMPool.java
 0c1e7866c0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 27a1bd8f25 
  standalone-metastore/src/main/resources/package.jdo b271e026e9 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 9f92bf6be2 


Diff: https://reviews.apache.org/r/63346/diff/11/

Changes: https://reviews.apache.org/r/63346/diff/10-11/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-07 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review190361
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 9471 (patched)


Atleast this should be documented in the config description. Launching 4 
AMs and not using them is also wasteful. On poorly configured cluster this can 
have adverse effects (default AMs taking up space of LLAP daemons). IMHO for 
better OOTB experience making it configurable is better option. We can ship 
with default 4 but if someone thinks 4 is wasteful then they have option of 
using default pool and less resources.


- Prasanth_J


On Nov. 7, 2017, 3:09 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Nov. 7, 2017, 3:09 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 305e9dc2ed 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
>  012361a186 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql f9b0765916 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 054978e3e4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql a13d976e37 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1ceb7239eb 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 03ee4cfd64 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 06648540f0 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql a13d976e37 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql a94985a63e 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql e80e6124fc 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> be1bb1eee0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 826ae565c1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 9954c249d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 169991c639 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 3415a23dec 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 8708f2a0b4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> 04c9e4413d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 0347e914bc 
>   ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2f314a6a8d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 300ba7281d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7334a0c9fa 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f8ebc124c4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMPool.java
>  0c1e7866c0 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  27a1bd8f25 
>   standalone-metastore/src/main/resources/package.jdo b271e026e9 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 9f92bf6be2 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/10/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18003) add explicit jdbc connection string args for mappings

2017-11-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18003:
---

 Summary: add explicit jdbc connection string args for mappings
 Key: HIVE-18003
 URL: https://issues.apache.org/jira/browse/HIVE-18003
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


1) Force using unmanaged/containers execution.
2) Optional - specify pool name (config setting to gate this, disabled by 
default?).
In phase 2 (or 4?) we might allow #2 to be used by a user to choose between 
multiple mappings if they have multiple pools they could be mapped to (i.e. to 
change the ordering essentially). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18004) investigate deriving app name from JDBC connection for pool mapping

2017-11-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18004:
---

 Summary: investigate deriving app name from JDBC connection for 
pool mapping
 Key: HIVE-18004
 URL: https://issues.apache.org/jira/browse/HIVE-18004
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


There are some client info fields that popular apps (Tableau, etc) might 
populate; this might allow us to map queries to pools based on an application 
used. Need to take a look (see the doc for an example API we might look into)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18002) add group support for pool mappings

2017-11-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-18002:
---

 Summary: add group support for pool mappings
 Key: HIVE-18002
 URL: https://issues.apache.org/jira/browse/HIVE-18002
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-07 Thread Sergey Shelukhin 


> On Nov. 7, 2017, 5:13 p.m., Prasanth_J wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 9471 (patched)
> > 
> >
> > Why query parallelism is set to 4 here?

hmm... what should it be set to? I didn't want to make it configurable, because 
if you modify configs for this you might as well change the pool instead.
1 may create poor out of the box experience in an unobvious manner (one query 
at a time for unknown reason), at least with 4 if something is wrong (4 is too 
much) it's more obvious.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review190340
---


On Nov. 7, 2017, 3:09 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Nov. 7, 2017, 3:09 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 305e9dc2ed 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
>  012361a186 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql f9b0765916 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 054978e3e4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql a13d976e37 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1ceb7239eb 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 03ee4cfd64 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 06648540f0 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql a13d976e37 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql a94985a63e 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql e80e6124fc 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> be1bb1eee0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 826ae565c1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 9954c249d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 169991c639 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 3415a23dec 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 8708f2a0b4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> 04c9e4413d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 0347e914bc 
>   ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2f314a6a8d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 300ba7281d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7334a0c9fa 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f8ebc124c4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMPool.java
>  0c1e7866c0 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  27a1bd8f25 
>   standalone-metastore/src/main/resources/package.jdo b271e026e9 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 9f92bf6be2 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/10/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-18001) InvalidObjectException while creating Primary Key constraint on partition key column

2017-11-07 Thread Nita Dembla (JIRA) 
Nita Dembla created HIVE-18001:
--

 Summary: InvalidObjectException while creating Primary Key 
constraint on partition key column
 Key: HIVE-18001
 URL: https://issues.apache.org/jira/browse/HIVE-18001
 Project: Hive
  Issue Type: Bug
Reporter: Nita Dembla



{code}
hive> show create table inventory;
OK
CREATE TABLE `inventory`(
  `inv_item_sk` bigint,
  `inv_warehouse_sk` bigint,
  `inv_quantity_on_hand` int)
PARTITIONED BY (
  `inv_date_sk` bigint)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  
'hdfs://ctr-e134-1499953498516-233086-01-02.hwx.site:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_1000.db/inventory'
TBLPROPERTIES (
  'transient_lastDdlTime'='1508284425')
Time taken: 0.25 seconds, Fetched: 16 row(s)

hive> alter table inventory add constraint pk_in primary key (inv_date_sk, 
inv_item_sk, inv_warehouse_sk) disable novalidate rely;
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:Parent 
column not found: inv_date_sk)

{code}

Exception from the log
{code}
2017-11-07T18:17:50,516 ERROR [d4ed6f97-20ea-4bc8-a046-b0646f483a20 main] 
exec.DDLTask: Failed
org.apache.hadoop.hive.ql.metadata.HiveException: 
InvalidObjectException(message:Parent column not found: inv_date_sk)
at 
org.apache.hadoop.hive.ql.metadata.Hive.addPrimaryKey(Hive.java:4668) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.DDLTask.addConstraints(DDLTask.java:4356) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:413) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:206) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2276) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1906) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1623) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1362) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1352) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:827) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:765) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:692) 
~[hive-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_112]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_112]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at org.apache.hadoop.util.RunJar.run(RunJar.java:233) 
~[hadoop-common-2.7.3.2.6.2.0-205.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:148) 
~[hadoop-common-2.7.3.2.6.2.0-205.jar:?]
Caused by: org.apache.hadoop.hive.metastore.api.InvalidObjectException: Parent 
column not found: inv_date_sk
at 
org.apache.hadoop.hive.metastore.ObjectStore.addPrimaryKeys(ObjectStore.java:4190)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.ObjectStore.addPrimaryKeys(ObjectStore.java:4163)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_112]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_112]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)

[jira] [Created] (HIVE-18000) LazySimple Date serialization is very expensive

2017-11-07 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-18000:


 Summary: LazySimple Date serialization is very expensive
 Key: HIVE-18000
 URL: https://issues.apache.org/jira/browse/HIVE-18000
 Project: Hive
  Issue Type: Bug
  Components: Serializers/Deserializers
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


This is follow up for HIVE-17417 for date serialization. Refer HIVE-17417 for 
more info. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Integrating Yetus with Precommit job

2017-11-07 Thread Alan Gates 
I’ve put some feedback in HIVE-17995.  17996 and 17997 look good.  I’ll
commit them once the tests run.

I think you’ll need to do similar patches for storage-api, as it is also
not connected to the hive pom anymore.

Alan.

On Tue, Nov 7, 2017 at 6:17 AM, Adam Szita  wrote:

> Thanks for all the replies.
>
> Vihang: Good idea on making everything green before turning this on. For
> this purpose I've filed a couple of jiras:
> -HIVE-17995  Run
> checkstyle on standalone-metastore module with proper configuration
> -HIVE-17996  Fix ASF
> headers
> -HIVE-17997  Add rat
> plugin and configuration to standalone metastore pom
>
> Sahil: there is an umbrella jira (HIVE-13503
> ) for test improvements,
> the Yetus integration itself is also a subtask of it. I think any further
> improvements on what Yetus features we want to enable should go here too.
>
> Adam
>
>
>
>
>

Re: Integrating Yetus with Precommit job

2017-11-07 Thread Xuefu Zhang 
+1 on the ideas.

On Tue, Nov 7, 2017 at 6:17 AM, Adam Szita  wrote:

> Thanks for all the replies.
>
> Vihang: Good idea on making everything green before turning this on. For
> this purpose I've filed a couple of jiras:
> -HIVE-17995  Run
> checkstyle on standalone-metastore module with proper configuration
> -HIVE-17996  Fix ASF
> headers
> -HIVE-17997  Add rat
> plugin and configuration to standalone metastore pom
>
> Sahil: there is an umbrella jira (HIVE-13503
> ) for test improvements,
> the Yetus integration itself is also a subtask of it. I think any further
> improvements on what Yetus features we want to enable should go here too.
>
> Adam
>
>
>
> On 6 November 2017 at 22:02, Sahil Takiar  wrote:
>
> > +1 - think this will be a great addition to Hive. Helping us catch issues
> > earlier, keeping the Hive code cleaner, etc. Getting the basic Yetus
> checks
> > to work seems like a great start, do we have follow JIRAs to get more
> YETUS
> > tests integrated - e.g. FindBugs?
> >
> > On Mon, Nov 6, 2017 at 10:29 AM, Vihang Karajgaonkar <
> vih...@cloudera.com>
> > wrote:
> >
> > > Thanks Adam for this work. This is definitely useful and a good
> addition
> > to
> > > our test infrastructure.
> > >
> > > Can we fix the existing issues pointed by Yetus (in a separate JIRA) so
> > > that we have a +1 from yetus on the current code? Once that is done,
> > > committers can help keep it green as they review patches and merge it.
> > >
> > > Thanks,
> > > Vihang
> > >
> > > On Mon, Nov 6, 2017 at 9:04 AM, Thejas Nair 
> > wrote:
> > >
> > > > +1
> > > > Yes, I think this can help us catch many issues early on, it will be
> > very
> > > > useful!
> > > >
> > > >
> > > > On Mon, Nov 6, 2017 at 7:43 AM, Adam Szita 
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > As a next step of test subsystem improvements we would like to have
> > the
> > > > > Yetus check integrated with the ptest framework. This means that
> > > > whenever a
> > > > > new patch is uploaded - along with the already existing Precommit
> > test
> > > > run
> > > > > - Hive's Yetus patch check script would be triggered also. This
> > script
> > > > runs
> > > > > checkstyle, findbugs, ASF license check, etc with and without the
> > > > submitted
> > > > > patch applied and reports the diffs (i.e. how many checkstyle
> > problems
> > > > does
> > > > > the patch introduce).
> > > > >
> > > > > It would be executed parallel to the ptest test execution and
> report
> > > back
> > > > > the results as a (another) jira comment to the issue in question.
> > > > > In the last days I've been working on this (HIVE-16748) and a patch
> > is
> > > > > ready to make this happen. A sample Yetus result comment is
> available
> > > at
> > > > > https://issues.apache.org/jira/browse/HIVE-16748?
> > > > > focusedCommentId=16218616=com.atlassian.jira.
> > > > > plugin.system.issuetabpanels:comment-tabpanel#comment-16218616
> > > > >
> > > > > We think this would be a useful tool for us developers and would
> like
> > > to
> > > > go
> > > > > ahead with this change, but we're also curious about your input in
> > this
> > > > > matter. Please let us know what you think about this change.
> > > > >
> > > > > Thanks,
> > > > > Adam
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sahil Takiar
> > Software Engineer at Cloudera
> > takiar.sa...@gmail.com | (510) 673-0309
> >
>

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-07 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review190340
---




metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql
Line 115 (original), 115 (patched)


I think Harish mentioned about POOL_ID to be BIGINT. RP_ID should also be 
BIGINT



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java
Lines 49 (patched)


follow up jira?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 9471 (patched)


Why query parallelism is set to 4 here?


- Prasanth_J


On Nov. 7, 2017, 3:09 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Nov. 7, 2017, 3:09 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 305e9dc2ed 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
>  012361a186 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql f9b0765916 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 054978e3e4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql a13d976e37 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1ceb7239eb 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 03ee4cfd64 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 06648540f0 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql a13d976e37 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql a94985a63e 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql e80e6124fc 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> be1bb1eee0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 826ae565c1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 9954c249d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 169991c639 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 3415a23dec 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 8708f2a0b4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> 04c9e4413d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 0347e914bc 
>   ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2f314a6a8d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 300ba7281d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7334a0c9fa 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f8ebc124c4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMPool.java
>  0c1e7866c0 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  27a1bd8f25 
>   standalone-metastore/src/main/resources/package.jdo b271e026e9 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 9f92bf6be2 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/10/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-17999) Remove hadoop3 hack in TestJdbcWithLocalClusterSpark and TestMultiSessionsHS2WithLocalClusterSpark after Spark supports Hadoop3

2017-11-07 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-17999:
---

 Summary: Remove hadoop3 hack in TestJdbcWithLocalClusterSpark and 
TestMultiSessionsHS2WithLocalClusterSpark after Spark supports Hadoop3
 Key: HIVE-17999
 URL: https://issues.apache.org/jira/browse/HIVE-17999
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu


Currently Spark hasn't supported Hadoop3 since it's blocked by Hive to support 
Hadoop3 so Hive takes the workaround to get HoS tests to pass (see 
TestJdbcWithLocalClusterSpark and TestMultiSessionsHS2WithLocalClusterSpark). 

SPARK-18673 is to enable the support of Hadoop3. After the work is done, we 
should upgrade Spark version dependency and remove such hack in these two tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17998) Use FastDateFormat instead of SimpleDateFormat for TimestampWritable

2017-11-07 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-17998:
--

 Summary: Use FastDateFormat instead of SimpleDateFormat for 
TimestampWritable
 Key: HIVE-17998
 URL: https://issues.apache.org/jira/browse/HIVE-17998
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Affects Versions: 3.0.0, 2.4.0
Reporter: BELUGA BEHR
Priority: Trivial


Currently Hive is using this ThreadLocal/SimpleDateFormat setup to work around 
the thread-safety limitations of SimpleDateFormat.

Let us simply drink the Apache Commons champagne and use thread-safe 
{{org.apache.commons.lang.time.FastDateFormat}} instead.

{code:java|title=org.apache.hadoop.hive.serde2.io.TimestampWritable}
  private static final ThreadLocal threadLocalDateFormat =
  new ThreadLocal() {
@Override
protected DateFormat initialValue() {
  return new SimpleDateFormat("-MM-dd HH:mm:ss");
}
  };
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Integrating Yetus with Precommit job

2017-11-07 Thread Adam Szita 
Thanks for all the replies.

Vihang: Good idea on making everything green before turning this on. For
this purpose I've filed a couple of jiras:
-HIVE-17995  Run
checkstyle on standalone-metastore module with proper configuration
-HIVE-17996  Fix ASF
headers
-HIVE-17997  Add rat
plugin and configuration to standalone metastore pom

Sahil: there is an umbrella jira (HIVE-13503
) for test improvements,
the Yetus integration itself is also a subtask of it. I think any further
improvements on what Yetus features we want to enable should go here too.

Adam



On 6 November 2017 at 22:02, Sahil Takiar  wrote:

> +1 - think this will be a great addition to Hive. Helping us catch issues
> earlier, keeping the Hive code cleaner, etc. Getting the basic Yetus checks
> to work seems like a great start, do we have follow JIRAs to get more YETUS
> tests integrated - e.g. FindBugs?
>
> On Mon, Nov 6, 2017 at 10:29 AM, Vihang Karajgaonkar 
> wrote:
>
> > Thanks Adam for this work. This is definitely useful and a good addition
> to
> > our test infrastructure.
> >
> > Can we fix the existing issues pointed by Yetus (in a separate JIRA) so
> > that we have a +1 from yetus on the current code? Once that is done,
> > committers can help keep it green as they review patches and merge it.
> >
> > Thanks,
> > Vihang
> >
> > On Mon, Nov 6, 2017 at 9:04 AM, Thejas Nair 
> wrote:
> >
> > > +1
> > > Yes, I think this can help us catch many issues early on, it will be
> very
> > > useful!
> > >
> > >
> > > On Mon, Nov 6, 2017 at 7:43 AM, Adam Szita  wrote:
> > >
> > > > Hi all,
> > > >
> > > > As a next step of test subsystem improvements we would like to have
> the
> > > > Yetus check integrated with the ptest framework. This means that
> > > whenever a
> > > > new patch is uploaded - along with the already existing Precommit
> test
> > > run
> > > > - Hive's Yetus patch check script would be triggered also. This
> script
> > > runs
> > > > checkstyle, findbugs, ASF license check, etc with and without the
> > > submitted
> > > > patch applied and reports the diffs (i.e. how many checkstyle
> problems
> > > does
> > > > the patch introduce).
> > > >
> > > > It would be executed parallel to the ptest test execution and report
> > back
> > > > the results as a (another) jira comment to the issue in question.
> > > > In the last days I've been working on this (HIVE-16748) and a patch
> is
> > > > ready to make this happen. A sample Yetus result comment is available
> > at
> > > > https://issues.apache.org/jira/browse/HIVE-16748?
> > > > focusedCommentId=16218616=com.atlassian.jira.
> > > > plugin.system.issuetabpanels:comment-tabpanel#comment-16218616
> > > >
> > > > We think this would be a useful tool for us developers and would like
> > to
> > > go
> > > > ahead with this change, but we're also curious about your input in
> this
> > > > matter. Please let us know what you think about this change.
> > > >
> > > > Thanks,
> > > > Adam
> > > >
> > >
> >
>
>
>
> --
> Sahil Takiar
> Software Engineer at Cloudera
> takiar.sa...@gmail.com | (510) 673-0309
>

[jira] [Created] (HIVE-17997) Add rat plugin and configuration to standalone metastore pom

2017-11-07 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-17997:
-

 Summary: Add rat plugin and configuration to standalone metastore 
pom
 Key: HIVE-17997
 URL: https://issues.apache.org/jira/browse/HIVE-17997
 Project: Hive
  Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17996) Fix ASF headers

2017-11-07 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-17996:
-

 Summary: Fix ASF headers
 Key: HIVE-17996
 URL: https://issues.apache.org/jira/browse/HIVE-17996
 Project: Hive
  Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita


Yetus check reports some ASF header related issues in Hive code. Let's fix them 
up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17995) Run checkstyle on standalone-metastore module with proper configuration

2017-11-07 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-17995:
-

 Summary: Run checkstyle on standalone-metastore module with proper 
configuration
 Key: HIVE-17995
 URL: https://issues.apache.org/jira/browse/HIVE-17995
 Project: Hive
  Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita


Maven module standalone-metastore is obviously not connected to Hive root pom, 
therefore if someone (or an automated Yetus check) runs {{mvn checkstyle}} it 
will not consider Hive-specific checkstyle settings (e.g. validates row lengths 
against 80, not 100)

We need to make sure standalone-metastore pom has the proper checkstyle 
configuration



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

2017-11-07 Thread Adam Szita via Review Board 


> On Nov. 6, 2017, 6:51 p.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 263-264 (original), 275-278 (patched)
> > 
> >
> > Shouldn't the partValues contain only the values from partBatch here?

Good catch, thanks!


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/#review190193
---


On Nov. 7, 2017, 10:22 a.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63528/
> ---
> 
> (Updated Nov. 7, 2017, 10:22 a.m.)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17969
> https://issues.apache.org/jira/browse/HIVE-17969
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactoring alter table code to use batching of partitions when calling the 
> heavy removeUnusedColumnDescriptor method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ccadac1ada6aaae884ab39f5d99e91b8c542404e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7334a0c9fa87b1fe5b4f6c9d2073a477bc0f11ad 
> 
> 
> Diff: https://reviews.apache.org/r/63528/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

2017-11-07 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/
---

(Updated Nov. 7, 2017, 10:22 a.m.)


Review request for hive, Peter Vary and Barna Zsombor Klara.


Changes
---

Adding missing null-check. Correcting issue pointed out by Vihang


Bugs: HIVE-17969
https://issues.apache.org/jira/browse/HIVE-17969


Repository: hive-git


Description
---

Refactoring alter table code to use batching of partitions when calling the 
heavy removeUnusedColumnDescriptor method


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 ccadac1ada6aaae884ab39f5d99e91b8c542404e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 7334a0c9fa87b1fe5b4f6c9d2073a477bc0f11ad 


Diff: https://reviews.apache.org/r/63528/diff/2/

Changes: https://reviews.apache.org/r/63528/diff/1-2/


Testing
---


Thanks,

Adam Szita

Re: Review Request 63346: HIVE-17902 add a notions of default pool and unmanaged mapping

2017-11-07 Thread Harish Jaiprakash 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63346/#review190280
---




metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql
Line 115 (original), 115 (patched)


POOL_ID here should also become BIGINT.


- Harish Jaiprakash


On Nov. 7, 2017, 8:39 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63346/
> ---
> 
> (Updated Nov. 7, 2017, 8:39 a.m.)
> 
> 
> Review request for hive, Harish Jaiprakash and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 305e9dc2ed 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersWorkloadManager.java
>  012361a186 
>   metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql f9b0765916 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql 054978e3e4 
>   metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql c1578fcda9 
>   metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql a13d976e37 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1ceb7239eb 
>   metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql 03ee4cfd64 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 06648540f0 
>   metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql a13d976e37 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql a94985a63e 
>   metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql e80e6124fc 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> be1bb1eee0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 826ae565c1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java a1b7cfb158 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 
> 9954c249d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
> 169991c639 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerFederation.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  f1258bab17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
>  9df1b82d62 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 3415a23dec 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0263df0d04 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 8708f2a0b4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 69a1d63f93 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterResourcePlanDesc.java 
> 04c9e4413d 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
> 0347e914bc 
>   ql/src/test/queries/clientpositive/resourceplan.q f1be695cb8 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 2f314a6a8d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 300ba7281d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7334a0c9fa 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  f8ebc124c4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMPool.java
>  0c1e7866c0 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  27a1bd8f25 
>   standalone-metastore/src/main/resources/package.jdo b271e026e9 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 9f92bf6be2 
> 
> 
> Diff: https://reviews.apache.org/r/63346/diff/10/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>