[jira] [Created] (HIVE-24742) Support router path or view fs path in Hive table location

[Aihua Xu (Jira)]

Aihua Xu created HIVE-24742:
---

 Summary: Support router path or view fs path in Hive table location
 Key: HIVE-24742
 URL: https://issues.apache.org/jira/browse/HIVE-24742
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.1.2
Reporter: Aihua Xu
Assignee: Aihua Xu


In 
[FileUtils.java|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L747],
 equalsFileSystem function checks the base URL to determine if source and 
destination are on the same cluster and decides copy or move the data. That 
will not work for viewfs or router base file system since viewfs://ns-default/a 
and viewfs://ns-default/b may be on different physical clusters.

FileSystem in HDFS supports resolvePath() function to resolve to the physical 
path. We can support viewfs and router through such function.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24171) Support HDFS reads from observer NameNodes

[Aihua Xu (Jira)]

Aihua Xu created HIVE-24171:
---

 Summary: Support HDFS reads from observer NameNodes
 Key: HIVE-24171
 URL: https://issues.apache.org/jira/browse/HIVE-24171
 Project: Hive
  Issue Type: New Feature
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HDFS-12943 introduces the consistent reads from observer NameNodes which can 
boost the read performance and reduces the overloads on active NameNodes.

To take advantage of this feature, the clients are required to make a msync() 
call after writing the files or before reading the files since observer 
NameNodes could have the stale data for a small window. 

Hive needs to make msync() call to HDFS in some places, e.g., 1) after 
generating the plan files - map.xml and reduce.xml so they can get used later 
by executors; 2) after the intermediate files are generated so they can get 
used by later stages or HS2. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-21122) Support Yarn resource profile in Hive

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-21122:
---

 Summary: Support Yarn resource profile in Hive
 Key: HIVE-21122
 URL: https://issues.apache.org/jira/browse/HIVE-21122
 Project: Hive
  Issue Type: New Feature
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu


Resource profile is a new feature supported in Yarn 3.1.0 (see YARN-3926). This 
would allow Yarn to allocate other resources like GPU/FPGA, in addition to 
memory and vcores. This would be a nice feature to support in Hive. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20861) Pass queryId as the client CallerContext to Spark

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20861:
---

 Summary: Pass queryId as the client CallerContext to Spark 
 Key: HIVE-20861
 URL: https://issues.apache.org/jira/browse/HIVE-20861
 Project: Hive
  Issue Type: Improvement
Reporter: Aihua Xu
Assignee: Aihua Xu


SPARK-16759 exposes a way for the client to pass the client CallerContext such 
as QueryId. For better debug, hive should pass queryId to spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20745) qtest-druid build is failing

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20745:
---

 Summary: qtest-druid build is failing
 Key: HIVE-20745
 URL: https://issues.apache.org/jira/browse/HIVE-20745
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 4.0.0
Reporter: Aihua Xu


qtest-druild build throws the following exception. Seems we are missing avro 
dependency in pom.xml.

{noformat}
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[9,31]
 package org.apache.avro.message does not exist
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[10,31]
 package org.apache.avro.message does not exist
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[11,31]
 package org.apache.avro.message does not exist
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[22,24]
 cannot find symbol
[ERROR]   symbol:   class BinaryMessageEncoder
[ERROR]   location: class org.apache.hive.kafka.Wikipedia
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[25,24]
 cannot find symbol
[ERROR]   symbol:   class BinaryMessageDecoder
[ERROR]   location: class org.apache.hive.kafka.Wikipedia
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[31,17]
 cannot find symbol
[ERROR]   symbol:   class BinaryMessageDecoder
[ERROR]   location: class org.apache.hive.kafka.Wikipedia
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[39,63]
 cannot find symbol
[ERROR]   symbol:   class SchemaStore
[ERROR]   location: class org.apache.hive.kafka.Wikipedia
[ERROR] 
/Users/aihuaxu/workspaces/hive-workspace/apache/hive/itests/qtest-druid/src/main/java/org/apache/hive/kafka/Wikipedia.java:[39,17]
 cannot find symbol
[ERROR]   symbol:   class BinaryMessageDecoder
[ERROR]   location: class org.apache.hive.kafka.Wikipedia
[ERROR] -> [Help 1]
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207983
---


Ship it!




Ship It!

- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedSto

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207981
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Line 200 (original), 200 (patched)
<https://reviews.apache.org/r/68395/#comment291704>

Seems it can be static since it doesn't refer to any variables outside. 
Maybe I miss anything?


- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207920
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Line 376 (original), 416 (patched)
<https://reviews.apache.org/r/68395/#comment291563>

Why some functions like extractSqlString() not moving into 
MetastoreDirectSqlUtils class, but extractSqlLong() gets moved? Can we make it 
consistent?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 547 (patched)
<https://reviews.apache.org/r/68395/#comment291564>

Better add comments for this public function.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 886 (patched)
<https://reviews.apache.org/r/68395/#comment291561>

This function is a little confusing actually. Since we are not used 
anywhere else, probably we don't need to create such function seperately.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Lines 200 (patched)
<https://reviews.apache.org/r/68395/#comment291568>

This function can be static and probably gives the name find() as well? 
findUtil is a little confusing to me.


- Aihua Xu


On Aug. 24, 2018, 6:15 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 24, 2018, 6:15 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
&g

[jira] [Created] (HIVE-20345) Drop database may hang by the change in HIVE-11258

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20345:
---

 Summary: Drop database may hang by the change in HIVE-11258
 Key: HIVE-20345
 URL: https://issues.apache.org/jira/browse/HIVE-20345
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 2.0.0, 1.3.0
Reporter: Aihua Xu
Assignee: Aihua Xu


In HiveMetaStore.java drop_database_core function,  HIVE-11258 updates the 
startIndex from endIndex incorrectly inside {{if (tables != null && 
!tables.isEmpty())}} statement. If the tables get deleted before 
getTableObjectsByName() call, then returned table list is empty and startIndex 
won't get updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20331) Query with union all, lateral view and Join fails with "cannot find parent in the child operator"

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20331:
---

 Summary: Query with union all, lateral view and Join fails with 
"cannot find parent in the child operator"
 Key: HIVE-20331
 URL: https://issues.apache.org/jira/browse/HIVE-20331
 Project: Hive
  Issue Type: Bug
  Components: Physical Optimizer
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


The following query with Union, Lateral view and Join will fail during 
execution with the exception below.
{noformat}
create table t1(col1 int);
SELECT 1 AS `col1`
FROM t1
UNION ALL
  SELECT 2 AS `col1`
  FROM
(SELECT col1
 FROM t1
) x1
JOIN
  (SELECT col1
  FROM
(SELECT 
  Row_Number() over (PARTITION BY col1 ORDER BY col1) AS `col1`
FROM t1
) x2 lateral VIEW explode(map(10,1))`mapObj` AS `col2`, `col3`
  ) `expdObj`  
{noformat}

{noformat}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive internal 
error: cannot find parent in the child operator!
at 
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:509)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
{noformat}

After debugging, seems we have issues in GenMRFileSink1 class in which we are 
setting incorrect aliasToWork to the MapWork.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20079) Populate more accurate rawDataSize for parquet format

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20079:
---

 Summary: Populate more accurate rawDataSize for parquet format
 Key: HIVE-20079
 URL: https://issues.apache.org/jira/browse/HIVE-20079
 Project: Hive
  Issue Type: Improvement
  Components: File Formats
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Run the following queries and you will see the raw data for the table is 4 
(that is the number of fields) incorrectly. We need to populate correct data 
size so data can be split properly.
{noformat}
SET hive.stats.autogather=true;
CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET;
INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1');
DESC FORMATTED parquet_stats;
{noformat}

{noformat}
Table Parameters:
COLUMN_STATS_ACCURATE   true
numFiles1
numRows 2
rawDataSize 4
totalSize   373
transient_lastDdlTime   1530660523
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20053) Separate Hive Security from SessionState

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20053:
---

 Summary: Separate Hive Security from SessionState
 Key: HIVE-20053
 URL: https://issues.apache.org/jira/browse/HIVE-20053
 Project: Hive
  Issue Type: Improvement
  Components: Security
Affects Versions: 3.0.0
Reporter: Aihua Xu


Right now we have Hive security classes associated with SessionState. When 
HiveServer2 starts, the service session will initialize it and later each 
session will need to reinitialize it. Since such security configuration is on 
service level, we should move security info out out SessionState and make it 
Singleton so we can initialize it once.

And also, since SessionState.setupAuth()  - to setup authentication and 
authorization is not synchronized, we could run into concurrency issue if 
queries or meta operations are run within same session. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20037) Print root cause exception's toString() rather than getMessage()

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20037:
---

 Summary: Print root cause exception's toString() rather than 
getMessage()
 Key: HIVE-20037
 URL: https://issues.apache.org/jira/browse/HIVE-20037
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


When we run HoS job and if it fails for some errors, we are printing the 
exception message rather than exception toString(), for some exceptions, e.g., 
this java.lang.NoClassDefFoundError, we are missing the exception type 
information. 

{noformat}
Failed to execute Spark task Stage-1, with exception 
'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client 
for Spark session cf054497-b073-4327-a315-68c867ce3434: 
org/apache/spark/SparkConf)'
{noformat}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20027) TestRuntimeStats.testCleanup is flaky

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-20027:
---

 Summary: TestRuntimeStats.testCleanup is flaky
 Key: HIVE-20027
 URL: https://issues.apache.org/jira/browse/HIVE-20027
 Project: Hive
  Issue Type: Bug
Reporter: Aihua Xu


int deleted = objStore.deleteRuntimeStats(1);
assertEquals(1, deleted);

The testCleanup could fail if somehow there is GC pause before 
deleteRuntimeStats happens so actually 2 stats will get deleted rather than one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19948) HiveCli is not splitting the command by semicolon properly if quotes are inside the string

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19948:
---

 Summary: HiveCli is not splitting the command by semicolon 
properly if quotes are inside the string 
 Key: HIVE-19948
 URL: https://issues.apache.org/jira/browse/HIVE-19948
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 2.2.0
Reporter: Aihua Xu


HIVE-15297 tries to split the command by considering semicolon inside string, 
but it doesn't consider the case that quotes can also be inside string. 

For the following command {{insert into escape1 partition (ds='1', part='3') 
values ("abc' ");}}, it will fail with 
{noformat}
18/06/19 16:37:05 ERROR ql.Driver: FAILED: ParseException line 1:64 extraneous 
input ';' expecting EOF near ''
org.apache.hadoop.hive.ql.parse.ParseException: line 1:64 extraneous input ';' 
expecting EOF near ''
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:220)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:606)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1686)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1633)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1628)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19936) explain on a query failing in secure cluster whereas query itself works

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19936:
---

 Summary: explain on a query failing in secure cluster whereas 
query itself works
 Key: HIVE-19936
 URL: https://issues.apache.org/jira/browse/HIVE-19936
 Project: Hive
  Issue Type: Bug
  Components: Hooks
Reporter: Aihua Xu


On a secured cluster with Sentry integrated run the following queries

{noformat}
create table foobar (id int) partitioned by (val int);
explain alter table foobar add partition (val=50);
{noformat}

The explain query will fail with the following exception while the query itself 
works with no issue.

Error while compiling statement: FAILED: SemanticException No valid 
privileges{color}
 Required privilege( Table) not available in output privileges
 The required privileges: (state=42000,code=4)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19899) Support stored as JsonFile

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19899:
---

 Summary: Support stored as JsonFile 
 Key: HIVE-19899
 URL: https://issues.apache.org/jira/browse/HIVE-19899
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.0.0
 Environment: This is to add "stored as jsonfile" support for json file 
format. 
Reporter: Aihua Xu
Assignee: Aihua Xu






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19835) Flaky test: TestWorkloadManager.testAsyncSessionInitFailures

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19835:
---

 Summary: Flaky test: 
TestWorkloadManager.testAsyncSessionInitFailures
 Key: HIVE-19835
 URL: https://issues.apache.org/jira/browse/HIVE-19835
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Affects Versions: 4.0.0
Reporter: Aihua Xu


Sometimes this test fails with the following issue. Seems it's a flaky test.

{noformat}
Error Message
expected:<0> but was:<1>
Stacktrace
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures(TestWorkloadManager.java:1138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19747) "GRANT ALL TO USER" failed with NullPointerException

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19747:
---

 Summary: "GRANT ALL TO USER" failed with NullPointerException
 Key: HIVE-19747
 URL: https://issues.apache.org/jira/browse/HIVE-19747
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 2.1.0
Reporter: Aihua Xu


If you issue the command 'grant all to user abc', you will see the following 
NPE exception. Seems the type in hivePrivObject is not initialized.

{noformat}
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.isOwner(SQLAuthorizationUtils.java:265)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPrivilegesFromMetaStore(SQLAuthorizationUtils.java:212)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.GrantPrivAuthUtils.checkRequiredPrivileges(GrantPrivAuthUtils.java:64)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.GrantPrivAuthUtils.authorize(GrantPrivAuthUtils.java:50)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.grantPrivileges(SQLStdHiveAccessController.java:179)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.grantPrivileges(SQLStdHiveAccessControllerWrapper.java:70)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.grantPrivileges(HiveAuthorizerImpl.java:48)
at 
org.apache.hadoop.hive.ql.exec.DDLTask.grantOrRevokePrivileges(DDLTask.java:1123
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19496) Check untar folder

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19496:
---

 Summary: Check untar folder
 Key: HIVE-19496
 URL: https://issues.apache.org/jira/browse/HIVE-19496
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Aihua Xu
Assignee: Aihua Xu


We need to check untar folder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19328) Some error messages like "table not found" are printing to STDERR

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19328:
---

 Summary: Some error messages like "table not found" are printing 
to STDERR
 Key: HIVE-19328
 URL: https://issues.apache.org/jira/browse/HIVE-19328
 Project: Hive
  Issue Type: Sub-task
  Components: Logging
Affects Versions: 3.0.0
Reporter: Aihua Xu


In Driver class, we are printing the exceptions to the log file and to the 
console through LogHelper. 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L730

I can see the following exceptions in the stderr.
FAILED: SemanticException [Error 10001]: Table not found default.sample_07

If it's from HiveCli, that makes sense to print to console, while if it's 
beeline talking to HS2, then such log should go to HS2 log and beeline console. 
So we should differentiate these two scenarios.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19320) MapRedLocalTask is printing child log to stderr and stdout

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19320:
---

 Summary: MapRedLocalTask is printing child log to stderr and stdout
 Key: HIVE-19320
 URL: https://issues.apache.org/jira/browse/HIVE-19320
 Project: Hive
  Issue Type: Sub-task
  Components: Logging
Affects Versions: 3.0.0
Reporter: Aihua Xu


In this line, local child MR task is printing the logs to stderr and stdout. 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java#L341

stderr/stdout should capture the service running log rather than the query 
execution output. Those should be reasonable to go to HS2 log and propagate to 
beeline console. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19318) Improve Hive logging

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19318:
---

 Summary: Improve Hive logging
 Key: HIVE-19318
 URL: https://issues.apache.org/jira/browse/HIVE-19318
 Project: Hive
  Issue Type: Improvement
  Components: Logging
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Use this jira to track some potential improvements on hive logging. What I have 
noticed that some log entries may have incorrect log level, or may not show in 
the correct places, e.g., printing to the STDERR/STDOUT rather than the HS2 log 
file. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu via Review Board]

> On March 22, 2018, 11:03 a.m., Peter Vary wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 154-158 (original), 154-158 (patched)
> > <https://reviews.apache.org/r/66188/diff/2/?file=1984629#file1984629line154>
> >
> > It might be a good idea, to use this around our batching as well:
> > - DatabaseProduct.needsInBatching(dbType)
> > 
> > What do you think @Aihua?

Thanks Peter for reviewing. This is slightly different problem. In directSQL, 
some databases may not need batch, some do. While in DN, the limitation is in 
DN, so it applies to all the databases.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/#review199751
-------


On April 23, 2018, 10:51 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66188/
> ---
> 
> (Updated April 23, 2018, 10:51 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Yongzhi Chen.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
> 
> I'm breaking the query into multiple smaller queries and then we aggregate 
> the result together.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/alter_rename_table.q 53fb230cf6 
>   ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  997f5fdb88 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  125d5a79f2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  59749e4947 
> 
> 
> Diff: https://reviews.apache.org/r/66188/diff/3/
> 
> 
> Testing
> ---
> 
> Manual test has been done for large column of tables.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

(Updated April 23, 2018, 10:51 p.m.)


Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Changes
---

Address comments.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs (updated)
-

  ql/src/test/queries/clientpositive/alter_rename_table.q 53fb230cf6 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 997f5fdb88 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 125d5a79f2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 59749e4947 


Diff: https://reviews.apache.org/r/66188/diff/3/

Changes: https://reviews.apache.org/r/66188/diff/2-3/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu via Review Board]

> On April 17, 2018, 4:49 p.m., Yongzhi Chen wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 7730 (patched)
> > <https://reviews.apache.org/r/66188/diff/2/?file=1984630#file1984630line7731>
> >
> > Should you call addQueryAfterUse and closeAllQueries ? That's how do 
> > you release the resources held by the batch queries?

In this change, I reuse single query to perform multiple queries and then 
release the query using queryWrapper. I did check how these two are used. Seems 
it's misused in other places that whenever there is new query, we created a new 
Query object which is not necessary. We can improve the logic to reuse the same 
Query object. It doesn't seem to be necessary.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/#review201323
-------


On March 21, 2018, 6:57 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66188/
> ---
> 
> (Updated March 21, 2018, 6:57 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Yongzhi Chen.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
> 
> I'm breaking the query into multiple smaller queries and then we aggregate 
> the result together.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/alter_rename_table.q 2061850540 
>   ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  6ead20aeaf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  88d88ed4df 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  9f822564bd 
> 
> 
> Diff: https://reviews.apache.org/r/66188/diff/2/
> 
> 
> Testing
> ---
> 
> Manual test has been done for large column of tables.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

[jira] [Created] (HIVE-19223) Migrate negative test cases to use hive.cli.errors.ignore

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19223:
---

 Summary: Migrate negative test cases to use hive.cli.errors.ignore 
 Key: HIVE-19223
 URL: https://issues.apache.org/jira/browse/HIVE-19223
 Project: Hive
  Issue Type: Improvement
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu


Migrate the negative test cases to use hive.cli.errors.ignore properties so 
multiple negative tests can be grouped together. It will save test resources 
and execution times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19222) TestNegativeCliDriver tests are failing due to "java.lang.OutOfMemoryError: GC overhead limit exceeded"

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19222:
---

 Summary: TestNegativeCliDriver tests are failing due to 
"java.lang.OutOfMemoryError: GC overhead limit exceeded"
 Key: HIVE-19222
 URL: https://issues.apache.org/jira/browse/HIVE-19222
 Project: Hive
  Issue Type: Sub-task
Reporter: Aihua Xu


TestNegativeCliDriver tests are failing with OOM recently. Not sure why. I will 
try to increase the memory to test out.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19204:
---

 Summary: Detailed errors from some tasks are not displayed to the 
client because the tasks don't set exception when they fail
 Key: HIVE-19204
 URL: https://issues.apache.org/jira/browse/HIVE-19204
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


In TaskRunner.java, if the tasks have exception set, then the task result will 
have such exception set and Driver.java will get such details and display to 
the client. But some tasks don't set such exceptions so the client won't see 
such details unless you check the HS2 log.
  
{noformat}
  public void runSequential() {
int exitVal = -101;
try {
  exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
} catch (Throwable t) {
  if (tsk.getException() == null) {
tsk.setException(t);
  }
  LOG.error("Error in executeTask", t);
}
result.setExitVal(exitVal);
if (tsk.getException() != null) {
  result.setTaskError(tsk.getException());
}
  }
 {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19040) get_partitions_by_expr() implementation in HiveMetaStore causes backward incompatibility easily

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19040:
---

 Summary: get_partitions_by_expr() implementation  in HiveMetaStore 
causes backward incompatibility easily
 Key: HIVE-19040
 URL: https://issues.apache.org/jira/browse/HIVE-19040
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 2.0.0
Reporter: Aihua Xu


In the HiveMetaStore implementation of {{public PartitionsByExprResult 
get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an 
expression is serialized into byte array from the client side and passed 
through  PartitionsByExprRequest. Then HMS will deserialize back into the 
expression and filter the partitions by it.

Such partition filtering expression can contain various UDFs. If there are some 
changes to one of the UDFs between different Hive versions, HS2 on the older 
version will serialize the expression in old format which won't be able to be 
deserialized by HMS on the newer version.  One example of that is, GenericUDFIn 
class adds {{transient}}  to the field constantInSet which will cause such 
incompatibility.

One approach I'm thinking is, instead of converting the expression object to 
byte array, we can pass the expression string directly. 

 

 

  

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19018:
---

 Summary: beeline -e now requires semicolon even when used with 
query from command line
 Key: HIVE-19018
 URL: https://issues.apache.org/jira/browse/HIVE-19018
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}}, beeline 
console will wait for you to enter ';". It's a regression from the old 
behavior. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19010) Improve column stats update

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19010:
---

 Summary: Improve column stats update 
 Key: HIVE-19010
 URL: https://issues.apache.org/jira/browse/HIVE-19010
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


I'm seeing the column stats update could be inefficient. Use the subtasks of 
this Jira to track the improvements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

(Updated March 21, 2018, 6:57 p.m.)


Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Changes
---

Added unit tests and fix checkstyle.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs (updated)
-

  ql/src/test/queries/clientpositive/alter_rename_table.q 2061850540 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 6ead20aeaf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 88d88ed4df 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 9f822564bd 


Diff: https://reviews.apache.org/r/66188/diff/2/

Changes: https://reviews.apache.org/r/66188/diff/1-2/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 6ead20aeaf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 88d88ed4df 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 9f822564bd 


Diff: https://reviews.apache.org/r/66188/diff/1/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

[jira] [Created] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18986:
---

 Summary: Table rename will run java.lang.StackOverflowError in 
dataNucleus if the table contains large number of columns
 Key: HIVE-18986
 URL: https://issues.apache.org/jira/browse/HIVE-18986
 Project: Hive
  Issue Type: Improvement
  Components: Standalone Metastore
Reporter: Aihua Xu
Assignee: Aihua Xu
 Fix For: 3.0.0


If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2'.

 

2018-03-13 17:19:52,770 INFO 
org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: 
ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: db=default 
tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 17:20:00,495 
ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-200]: 
java.lang.StackOverflowError at 
org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at 
org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at 
org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18887) Improve preserving column stats for alter table commands

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18887:
---

 Summary: Improve preserving column stats for alter table commands
 Key: HIVE-18887
 URL: https://issues.apache.org/jira/browse/HIVE-18887
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


We are trying to preserve column stats for certain alter table commands, while 
seems that current generic approach which compare the old columns against the 
new columns and update for all the columns may not be efficient . e.g., if we 
just rename the table, we should be able to update the name itself. COL_STATS 
table somehow contains DB_Name and Table_Name. If those tables don't have these 
columns, certain commands don't even need to update these tables. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/#review197073
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2980 (patched)
<https://reviews.apache.org/r/65500/#comment277184>

I'm confused that we need this configuration. So we need to manually verify 
if the data is overflow and if there is overflow, we turn that on? 

Even we want to provide both CHECKED and UNCHECKED, I think we need to fail 
the query if there is overflow for unchecked implementation, otherwise, the 
user may get incorrect result. 

Of course, to me, I think we should just have CHECKED implementation, that 
is, we should always handle overflow internally.

BTW: how much performance impact will it be? And why do we  have big 
performance impact from the CHECKED implementation (sorry, I don't follow 
exactly the vecorization work)?



ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt
Lines 131 (patched)
<https://reviews.apache.org/r/65500/#comment277183>

Can we remove #ELSE from here and some other places below? Seems not needed.


- Aihua Xu


On Feb. 7, 2018, 7:13 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65500/
> ---
> 
> (Updated Feb. 7, 2018, 7:13 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.
> 
> 
> Bugs: HIVE-18421
> https://issues.apache.org/jira/browse/HIVE-18421
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
> b5011c3adcedf8974d3241994733e0021a851cbd 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
> cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
> 3e955578933dd7990939865527c3bd11023b3a90 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
> f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
> e95baa6199e138a4e0c009e62ce495b626e5909c 
>   ql/src/gen/vectorization/TestTemplates/TestClass.txt 
> 62c58fb293fbe2d4d948c6a3409ee31466424a02 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  bbe78c8720e16163b642f54d27fdf6b65ba9850b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 8264e8ad285deac29424bd1cb0bf626436d47c75 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
>  75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
>  6b4d714c9a79a55593c4a4d254267a3035abb10f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  710165033627b33d9b238cc847dbac36c07ee5f6 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> af8552caa02f2896f393a5099abdb1ae5abd4c16 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
> e2a638da518a2071ff15b8da6899646ec45c832a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
> 99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
> 4e45788936559bbb7cfe65e9ffd083747b37dcc2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
> b1200e673e6b470b5fd1cc856270a6da615f16cb 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
>  ab6f6b79316818cac458390dc2d087091057c63b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
>  02dec659ce421eef06f924bb6973070878d57be3 
>   

Re: Review Request 65330: HIVE-18526 Backport HIVE-16886 to Hive 2

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65330/#review196397
---


Ship it!




Ship It!

- Aihua Xu


On Jan. 27, 2018, 12:05 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65330/
> ---
> 
> (Updated Jan. 27, 2018, 12:05 a.m.)
> 
> 
> Review request for hive, Aihua Xu, anishek, Andrew Sherman, Janaki Lahorani, 
> Sergio Pena, Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18526
> https://issues.apache.org/jira/browse/HIVE-18526
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18526 Backport HIVE-16886 to Hive 2
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> d16d2a3cd6a08217bb5b5b7bd9226f5e19f68994 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
> 69e8826f53c9278a7307817c8bfa4151f36631b9 
> 
> 
> Diff: https://reviews.apache.org/r/65330/diff/2/
> 
> 
> Testing
> ---
> 
> I wrote unit test which runs against stand-alone DB and verified that 
> everything is as expected.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-18550) Keep the hbase table name property as hbase.table.name

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18550:
---

 Summary: Keep the hbase table name property as hbase.table.name
 Key: HIVE-18550
 URL: https://issues.apache.org/jira/browse/HIVE-18550
 Project: Hive
  Issue Type: Sub-task
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


With hbase 2.0 support, I made some changes to the hbase table name property 
change in HIVE-18366 and HIVE-18202. By checking the logic, seems the change is 
not necessary since hbase.table.name is internal to hive hbase handler. We just 
need to map hbase.table.name to hbase.mapreduce.hfileoutputformat.table.name 
for HiveHFileOutputFormat. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-18366) Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the table name property

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18366:
---

 Summary: Update HBaseSerDe to use 
hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the 
table name property
 Key: HIVE-18366
 URL: https://issues.apache.org/jira/browse/HIVE-18366
 Project: Hive
  Issue Type: Sub-task
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HBase 2.0 changes the table name property to 
hbase.mapreduce.hfileoutputformat.table.name. HiveHFileOutputFormat is using 
the new property name while HiveHBaseTableOutputFormat is not. If we create the 
table as follows, HiveHBaseTableOutputFormat is used which still uses the old 
property hbase.table.name.

{noformat}
create table hbase_table2(key int, val string) stored by 
'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
('hbase.columns.mapping' = ':key,cf:val') tblproperties 
('hbase.mapreduce.hfileoutputformat.table.name' = 'positive_hbase_handler_bulk')
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18327) Remove the unnecessary HiveConf dependency for MiniHiveKdc

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18327:
---

 Summary: Remove the unnecessary HiveConf dependency for MiniHiveKdc
 Key: HIVE-18327
 URL: https://issues.apache.org/jira/browse/HIVE-18327
 Project: Hive
  Issue Type: Test
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu


MiniHiveKdc takes HiveConf as input parameter while it's not needed. Remove the 
unnecessary HiveConf.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18323:
---

 Summary: Vectorization: add the support of timestamp in 
VectorizedPrimitiveColumnReader
 Key: HIVE-18323
 URL: https://issues.apache.org/jira/browse/HIVE-18323
 Project: Hive
  Issue Type: Improvement
  Components: Vectorization
Affects Versions: 3.0.0
Reporter: Aihua Xu


{noformat}
CREATE TABLE `t1`(
  `ts` timestamp,
  `s1` string)
STORED AS PARQUET;

set hive.vectorized.execution.enabled=true;
SELECT * from t1 SORT BY s1;
{noformat}

This query will throw exception since timestamp is not supported here yet.

{noformat}
Caused by: java.io.IOException: java.io.IOException: Unsupported type: optional 
int96 ts
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18202) Automatically migrate hbase.table.name to hbase.mapreduce.hfileoutputformat.table.name for hbase-based table

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18202:
---

 Summary: Automatically migrate hbase.table.name to 
hbase.mapreduce.hfileoutputformat.table.name for hbase-based table
 Key: HIVE-18202
 URL: https://issues.apache.org/jira/browse/HIVE-18202
 Project: Hive
  Issue Type: Sub-task
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The property name for Hbase table mapping is changed from hbase.table.name to 
hbase.mapreduce.hfileoutputformat.table.name in HBase 2.

We can include such upgrade for existing hbase-based tables in DB upgrade 
script to automatically change such values.

For the new tables, the query will be like:

create table hbase_table(key int, val string) stored by 
'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
('hbase.columns.mapping' = ':key,cf:val') tblproperties 
('hbase.mapreduce.hfileoutputformat.table.name' = 'positive_hbase_handler_bulk')




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Apache Hive 2.3.2 Release Candidate 0

[Aihua Xu]

I built from the sources and ran a few tests. Checked gpg signatures.
Checked md5sum. Looks good. +1.

Thanks Sahil to put the release together.

Aihua

On Mon, Nov 13, 2017 at 8:43 AM, Sergio Pena 
wrote:

> +1
>
> I verified the release by doing the following:
> * checked the gpg signature
> * checked the md5 files
> * installed hive 2.3.2 in my local machine with hadoop 2.7.2 and run a few
> commands:
>   > show databases
>   > show tables
>   > insert into table values()
>   > select * from table
>   > select count(*) from table
> * checked the maven artifacts are correctly pulled by other components and
> run unit tests
> * checked that storage-api-2.4.0 is pulled
> * checked the release tag
> * checked the RELEASE_NOTES, NOTICE, LICENSE are correct
>
> The release is working correctly.
>
> Thanks Sahil for making this release.
> - Sergio
>
> On Thu, Nov 9, 2017 at 5:37 PM, Sahil Takiar 
> wrote:
>
> > Apache Hive 2.3.2 Release Candidate 0 is available here:
> > http://people.apache.org/~stakiar/hive-2.3.2/
> >
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1082/
> >
> > Source tag for RCN is at:https://github.com/apache/h
> ive/tree/release-2.3.2
> >
> > Voting will conclude in 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
>



-- 
Thanks
Aihua

[jira] [Created] (HIVE-18023) Redact the expression in lineage info

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18023:
---

 Summary: Redact the expression in lineage info
 Key: HIVE-18023
 URL: https://issues.apache.org/jira/browse/HIVE-18023
 Project: Hive
  Issue Type: Improvement
  Components: Logging
Affects Versions: 2.1.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Priority: Trivial


The query redactor is redacting the query itself while the expression shown in 
lineage info is not, which may still expose sensitive info. The following query

{{select customers.id, customers.name from customers where 
customers.addresses['shipping'].zip_code ='1234-5678-1234-5678';}} will have a 
log entry in lineage. The expression should also be redacted.

{noformat}
[HiveServer2-Background-Pool: Thread-43]: 
{"version":"1.0","user":"hive","timestamp":1510179280,"duration":40747,"jobIds":["job_1510150684172_0006"],"engine":"mr","database":"default","hash":"a2b4721a0935e3770d81649d24ab1cd4","queryText":"select
 customers.id, customers.name from customers where 
customers.addresses['shipping'].zip_code 
='---'","edges":[{"sources":[2],"targets":[0],"edgeType":"PROJECTION"},{"sources":[3],"targets":[1],"edgeType":"PROJECTION"},{"sources":[],"targets":[0,1],"expression":"(addresses['shipping'].zip_code
 = 
'1234-5678-1234-5678')","edgeType":"PREDICATE"}],"vertices":[{"id":0,"vertexType":"COLUMN","vertexId":"customers.id"},{"id":1,"vertexType":"COLUMN","vertexId":"customers.name"},{"id":2,"vertexType":"COLUMN","vertexId":"default.customers.id"},{"id":3,"vertexType":"COLUMN","vertexId":"default.customers.name"}]}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-18009) Multiple lateral view query is slow on hive on spark

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-18009:
---

 Summary: Multiple lateral view query is slow on hive on spark
 Key: HIVE-18009
 URL: https://issues.apache.org/jira/browse/HIVE-18009
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


When running the query with multiple lateral view, HoS is busy with the 
compilation. GenSparkUtils has an efficient implementation of getChildOperator 
when we have diamond hierarchy in operator trees (lateral view in this case) 
since the node may be visited multiple times.

{noformat}
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:442)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator(GenSparkUtils.java:438)
at 
org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils.getChildOperator

[jira] [Created] (HIVE-17999) Remove hadoop3 hack in TestJdbcWithLocalClusterSpark and TestMultiSessionsHS2WithLocalClusterSpark after Spark supports Hadoop3

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17999:
---

 Summary: Remove hadoop3 hack in TestJdbcWithLocalClusterSpark and 
TestMultiSessionsHS2WithLocalClusterSpark after Spark supports Hadoop3
 Key: HIVE-17999
 URL: https://issues.apache.org/jira/browse/HIVE-17999
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu


Currently Spark hasn't supported Hadoop3 since it's blocked by Hive to support 
Hadoop3 so Hive takes the workaround to get HoS tests to pass (see 
TestJdbcWithLocalClusterSpark and TestMultiSessionsHS2WithLocalClusterSpark). 

SPARK-18673 is to enable the support of Hadoop3. After the work is done, we 
should upgrade Spark version dependency and remove such hack in these two tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

(Updated Nov. 1, 2017, 10:20 p.m.)


Review request for hive.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs (updated)
-

  common/pom.xml 868e14dbc3 
  hbase-handler/pom.xml 7f57b77168 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f0 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa9 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed6562 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/rcfile/TestRCFileMapReduceInputFormat.java
 bb6c582d28 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c1 
  itests/hcatalog-unit/pom.xml bb6b10554f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f9 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514 
  itests/hive-minikdc/pom.xml 95d2614901 
  itests/hive-unit-hadoop2/pom.xml 339a194f2d 
  itests/hive-unit/pom.xml 14409832c8 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
dbfc23510c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b7 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithLocalClusterSpark.java
 cabddea5c6 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestMultiSessionsHS2WithLocalClusterSpark.java
 e3f9646b59 
  itests/qtest-accumulo/pom.xml 40d0a749c8 
  itests/qtest-spark/pom.xml a506f7f081 
  itests/qtest/pom.xml 02664f3792 
  itests/util/pom.xml e6dc09fb8f 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a89085 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925 
  llap-server/pom.xml 47a04cc310 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b6 
  metastore/pom.xml 04c6f47879 
  pom.xml 006e8f8611 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java 19b97e4cf2 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd 
  ql/src/test/results/clientnegative/external1.q.out 661d669206 
  ql/src/test/results/clientnegative/external2.q.out eb5518c017 
  serde/pom.xml 7419cfb17a 
  shims/0.23/pom.xml 3ff1d38776 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11d 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf4 
  shims/scheduler/pom.xml 0eadb69435 
  spark-client/pom.xml 784d908e4a 
  standalone-metastore/pom.xml 8df622f179 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a 
  testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 


Diff: https://reviews.apache.org/r/63394/diff/3/

Changes: https://reviews.apache.org/r/63394/diff/2-3/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189762
---



- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
> 2c37a51cf4 
>   shims/scheduler/pom.xml 0eadb69435 
>   spark-client/pom.xml 784d908e4a 
>   standalone-metastore/pom.xml 8df622f179 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
>  c10e36f94a 
>   testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 
> 
> 
> Diff: https://reviews.apache.org/r/63394/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189747
---




hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
Lines 119-120 (patched)
<https://reviews.apache.org/r/63394/#comment266991>

You mean we want to support NULL for Hbase table locations? I think we 
allow this 

`LOCATION` varchar(4000) DEFAULT NULL.

Seems we just need to correct the comments.



itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
Lines 106 (patched)
<https://reviews.apache.org/r/63394/#comment266962>

I think this will be needed on the cluster as well so hbase will use the 
right native library. Hbase has a doc for that but I can't find it right now.


- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/ma

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

(Updated Oct. 31, 2017, 9:56 p.m.)


Review request for hive.


Changes
---

Address comments.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs (updated)
-

  common/pom.xml 868e14dbc3 
  hbase-handler/pom.xml 7f57b77168 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f0 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa9 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed6562 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c1 
  itests/hcatalog-unit/pom.xml bb6b10554f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f9 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514 
  itests/hive-minikdc/pom.xml 95d2614901 
  itests/hive-unit-hadoop2/pom.xml 339a194f2d 
  itests/hive-unit/pom.xml 14409832c8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b7 
  itests/qtest-accumulo/pom.xml 40d0a749c8 
  itests/qtest-spark/pom.xml a506f7f081 
  itests/qtest/pom.xml 02664f3792 
  itests/util/pom.xml e6dc09fb8f 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a89085 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925 
  llap-server/pom.xml 47a04cc310 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b6 
  metastore/pom.xml 04c6f47879 
  pom.xml 006e8f8611 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd 
  ql/src/test/results/clientnegative/external1.q.out 661d669206 
  ql/src/test/results/clientnegative/external2.q.out eb5518c017 
  serde/pom.xml 7419cfb17a 
  shims/0.23/pom.xml 3ff1d38776 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11d 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf4 
  shims/scheduler/pom.xml 0eadb69435 
  spark-client/pom.xml 784d908e4a 
  standalone-metastore/pom.xml 8df622f179 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a 
  testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 


Diff: https://reviews.apache.org/r/63394/diff/2/

Changes: https://reviews.apache.org/r/63394/diff/1-2/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/pom.xml
> > Line 82 (original), 82 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872068#file1872068line82>
> >
> > scope=test?

Actually the code does have a reference to it. Not only for test.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 46 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872069#file1872069line46>
> >
> > Might be a good idea to extend DefaultHiveMetaHook to take advantage of 
> > other extension points.

I'm not very faimilar with the implementation for insert operation. So I will 
delay that to later jira.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872069#file1872069line57>
> >
> > Any reason to not do this in this patch?

Actually they are closed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
> > Line 742 (original), 742 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872083#file1872083line742>
> >
> > Do we need to retain this for 0.23? That support is long gone :)

I will leave this cleanup for later jira since seems other places also have 
0.23 support.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
> > Lines 106 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872088#file1872088line107>
> >
> > Is this test only or on cluster also we need to set up this property?

This is needed for the cluster as well so habase will pick up right native lib.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > pom.xml
> > Line 841 (original), 907 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872104#file1872104line907>
> >
> > I don't think we need to depend on hbase-sever. If this is only for 
> > unit tests, shall we add test

Actually we need for the production code.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > serde/pom.xml
> > Line 155 (original), 161 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872108#file1872108line161>
> >
> > Do we need this as well? Proabably not.

It's actually needed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > shims/0.23/pom.xml
> > Line 67 (original), 73 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872109#file1872109line73>
> >
> > This shall not be needed.

This is needed for getMiniDfs() call.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/pom.xml
> > Line 179 (original), 179 (patched)
> > <https://reviews.apache.org/r/63394/diff/1/?file=1872114#file1872114line179>
> >
> > Probably not needed.

It's actually needed.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189595
---


On Oct. 28, 2017, 11:54 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 28, 2017, 11:54 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc321be1e4008ba1575568bbc81032791 
>   hbase-handler/pom.xml 7f57b771686aeb09d4d58a4ae591110c10260519 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3abfd78bb4139eb471beeb0b66211f2a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b1a2654b8865d10645963cbe69eaa33 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f00d4a8ffe4a3649486d0dfe1e19ae68 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c258983337a63a27

Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

[Aihua Xu via Review Board]

 0eadb69435f5b8ad3a933d4f83954a844355e18e 
  spark-client/pom.xml 784d908e4a33077bf529e71a6b695ba43abbf827 
  standalone-metastore/pom.xml 8df622f1796aab885e486b0abc1f00088aca1626 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a5e94c760ba0472777cb62333519369 
  testutils/ptest2/src/main/resources/batch-exec.vm 
2d16ca3e8a06a1a4e7e515dccf657b31304d8559 


Diff: https://reviews.apache.org/r/63394/diff/1/


Testing
---


Thanks,

Aihua Xu

[jira] [Created] (HIVE-17870) Update NoDeleteRollingFileAppender to use Log4j2 api

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17870:
---

 Summary: Update NoDeleteRollingFileAppender to use Log4j2 api
 Key: HIVE-17870
 URL: https://issues.apache.org/jira/browse/HIVE-17870
 Project: Hive
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Aihua Xu


NoDeleteRollingFileAppender is still using log4jv1 api. Since we already moved 
to use log4j2 in hive, we better update to use log4jv2 as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Apache Hive 2.3.1 Release Candidate 0

[Aihua Xu]

Thanks Jesus. At mean time, can I commit the change to branch-2.3 or should
I wait for the vote?

Thanks,
Aihua

On Fri, Oct 20, 2017 at 11:56 AM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> Hi Sergio,
>
> Aihua pinged me about the issue this morning. Since vote is already going
> on, I would suggest to continue the vote.
> If vote passes, HIVE-17831 would be part of 2.3.2 even if it is committed
> to the branch right now (2.3.2 could be released in a few weeks if needed).
> If the vote does not pass, we can check it in before creating another RC
> and then it would be part of 2.3.1.
>
> Thanks,
> Jesús
>
>
>
> On 10/20/17, 11:44 AM, "Sergio Pena"  wrote:
>
> >Great, thanks Jesus for preparing this. Just one request, can we include
> >HIVE-17831 in the 2.3.1 release? Sentry has been waiting to bump the Hive
> >version to 2.x for a while, and HIVE-17831 fixes an issue that is required
> >to make it work.
> >
> >It's a small fix and is already part of 3.0 and 2.4.
> >
> >- Sergio
> >
> >On Thu, Oct 19, 2017 at 10:06 PM, Jesus Camacho Rodriguez <
> >jcama...@apache.org> wrote:
> >
> >> Apache Hive 2.3.1 Release Candidate 0 is available here:
> >> http://people.apache.org/~jcamacho/hive-2.3.1-rc0/
> >>
> >> Maven artifacts are available here:
> >> https://repository.apache.org/content/repositories/orgapachehive-1081/
> >>
> >> Source tag for RC0 is at:
> >> https://github.com/apache/hive/releases/tag/release-2.3.1-rc0/
> >>
> >> Voting will conclude in 72 hours.
> >>
> >> Hive PMC Members: Please test and vote.
> >>
> >> Thanks.
> >>
> >>
> >>
> >>
>
>


-- 
Thanks
Aihua

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

[Aihua Xu via Review Board]

> On Oct. 17, 2017, 11:25 p.m., Aihua Xu wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 261 (patched)
> > <https://reviews.apache.org/r/63043/diff/1/?file=1857700#file1857700line262>
> >
> > OK. I started to recall why we did that this way. I think we don't want 
> > to call query.close() in the subclass since when you return query.execute() 
> > actually DN doesn't get the result yet until you consume those results. 
> > 
> > That's why we want the caller to be responsible to call close() 
> > explictly and the caller knows that it has consumed all the data. 
> > 
> > The new implmentation will close the query object inside the subclass 
> > right?
> 
> Alexander Kolbasov wrote:
> Originally Query wasn't Closeable or AutoCLoseable, so you couldn't use 
> it anyway. Now it is and it could be possible to use Query directly, but this 
> raises a problem of handling exceptions on close.
> 
> Answering the second part of your question. The query now is closed 
> immediately after the use (in subclass that creates the query). Query results 
> can be used after query close, but I need to copy them into another 
> collection - that's why there are a bunch of changes like `return new 
> ArrayList<>(result)`.
> 
> Aihua Xu wrote:
> I see. I believe the original solution is to avoid the memory pressure on 
> HMS. With your approach, we are prefetching all the data and save in a 
> collection, while the original approach delays the data retrieval until it's 
> needed. 
> 
> Is that possible to preserve that logic but still let it close 
> automatically?
> 
> Alexander Kolbasov wrote:
> Note that we are not copying the data itself in the new ArrayList, we are 
> copying pointers. So the only extra overhead is the newly allocated array 
> which is mostly a short-living object and not a big deal. The prefetches are 
> present in the old code - I didn't add any new ones.
> 
> Doing what you suggest requires bigger code refactoring since 
> try-with-resource is based on try-blocks, so we need to significantly shuffle 
> the code around.

Maybe I didn't state clearly. When you call query.execute(), what I have 
noticed that DN actually didn't really execute the query. It will only return 
an object and wait until the result is getting consumed and DN will do the 
actual query against the database.

So for the following code, I remember query.execute() doesn't return any data 
for you at that time yet. Now if new ArrayList<> retrieves the data and it 
triggers the actual execution, then you are fetching all the data at a time 
(cause memory pressure); if it doesn't and if q.close() is called, then the 
List you returned to the parent is not accessible since the query is closed.

you will have either issue. That's what I remembered from the past. Can you 
check test failure TestHiveMetaTool.testExecuteJDOQL if it's related to this? 
Right now we are using directSQL access, so we may hide these DN issues from 
the tests.

private List sub() {

  try (queryWrapper q) {
  
 return new ArrayList<>((List) query.execute(dbName, tableName, 
partNameMatcher));
  }
}


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

[Aihua Xu via Review Board]

> On Oct. 17, 2017, 11:25 p.m., Aihua Xu wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 261 (patched)
> > <https://reviews.apache.org/r/63043/diff/1/?file=1857700#file1857700line262>
> >
> > OK. I started to recall why we did that this way. I think we don't want 
> > to call query.close() in the subclass since when you return query.execute() 
> > actually DN doesn't get the result yet until you consume those results. 
> > 
> > That's why we want the caller to be responsible to call close() 
> > explictly and the caller knows that it has consumed all the data. 
> > 
> > The new implmentation will close the query object inside the subclass 
> > right?
> 
> Alexander Kolbasov wrote:
> Originally Query wasn't Closeable or AutoCLoseable, so you couldn't use 
> it anyway. Now it is and it could be possible to use Query directly, but this 
> raises a problem of handling exceptions on close.
> 
> Answering the second part of your question. The query now is closed 
> immediately after the use (in subclass that creates the query). Query results 
> can be used after query close, but I need to copy them into another 
> collection - that's why there are a bunch of changes like `return new 
> ArrayList<>(result)`.

I see. I believe the original solution is to avoid the memory pressure on HMS. 
With your approach, we are prefetching all the data and save in a collection, 
while the original approach delays the data retrieval until it's needed. 

Is that possible to preserve that logic but still let it close automatically?


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

[Aihua Xu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 261 (patched)
<https://reviews.apache.org/r/63043/#comment265419>

OK. I started to recall why we did that this way. I think we don't want to 
call query.close() in the subclass since when you return query.execute() 
actually DN doesn't get the result yet until you consume those results. 

That's why we want the caller to be responsible to call close() explictly 
and the caller knows that it has consumed all the data. 

The new implmentation will close the query object inside the subclass right?


- Aihua Xu


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-17762) Exclude older jackson-annotation.jar from druid-handler shaded jar

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17762:
---

 Summary: Exclude older jackson-annotation.jar from druid-handler 
shaded jar
 Key: HIVE-17762
 URL: https://issues.apache.org/jira/browse/HIVE-17762
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


hive-druid-handler.jar is shading jackson core dependencies in hive-17468 but 
older versions are brought in from the transitive dependencies. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17699) Skip calling authValidator.checkPrivileges when there is nothing to get authorized

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17699:
---

 Summary: Skip calling authValidator.checkPrivileges when there is 
nothing to get authorized
 Key: HIVE-17699
 URL: https://issues.apache.org/jira/browse/HIVE-17699
 Project: Hive
  Issue Type: Improvement
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


For the command like "drop database if exists db1;" and the database db1 
doesn't exist, there will be nothing to get authorized. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17679) http-generic-click-jacking for WebHcat server

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17679:
---

 Summary: http-generic-click-jacking for WebHcat server
 Key: HIVE-17679
 URL: https://issues.apache.org/jira/browse/HIVE-17679
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


The web UIs do not include the "X-Frame-Options" header to prevent the pages 
from being framed from another site.
Reference:
https://www.owasp.org/index.php/Clickjacking
https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet
https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62736: HIVE-17544: Provide classname info for function authorization

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62736/#review186894
---




ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java
Line 259 (original), 273 (patched)
<https://reviews.apache.org/r/62736/#comment263771>

Other analyzer will call different constructurs here, so className will be 
null by default and we will only have className set for Functions.


- Aihua Xu


On Oct. 2, 2017, 9:10 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62736/
> ---
> 
> (Updated Oct. 2, 2017, 9:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17544: Provide classname info for function authorization
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> f01edf8156205ea452178f6539e5bbfe29742428 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 
> 820e4e2f67506e21efa0a1c7aaa37fb1c4b88f55 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 
> 4707c4d1bbc71e256b16511f48f8650c027773fe 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
> c5380750f23cc731d33354ea09858882b2364ce7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java
>  570571b274a090e2147d3ac485194082be12b813 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java
>  fb4c3206be3a92b3fd52826082fe103d5f445d0e 
> 
> 
> Diff: https://reviews.apache.org/r/62736/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 62736: HIVE-17544: Provide classname info for function authorization

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62736/
---

(Updated Oct. 2, 2017, 9:10 p.m.)


Review request for hive.


Repository: hive-git


Description
---

HIVE-17544: Provide classname info for function authorization


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
f01edf8156205ea452178f6539e5bbfe29742428 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 
820e4e2f67506e21efa0a1c7aaa37fb1c4b88f55 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 
4707c4d1bbc71e256b16511f48f8650c027773fe 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
c5380750f23cc731d33354ea09858882b2364ce7 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java
 570571b274a090e2147d3ac485194082be12b813 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java
 fb4c3206be3a92b3fd52826082fe103d5f445d0e 


Diff: https://reviews.apache.org/r/62736/diff/1/


Testing (updated)
---

Manually tested.


Thanks,

Aihua Xu

Review Request 62736: HIVE-17544: Provide classname info for function authorization

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62736/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-17544: Provide classname info for function authorization


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
f01edf8156205ea452178f6539e5bbfe29742428 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 
820e4e2f67506e21efa0a1c7aaa37fb1c4b88f55 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 
4707c4d1bbc71e256b16511f48f8650c027773fe 
  ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 
c5380750f23cc731d33354ea09858882b2364ce7 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java
 570571b274a090e2147d3ac485194082be12b813 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java
 fb4c3206be3a92b3fd52826082fe103d5f445d0e 


Diff: https://reviews.apache.org/r/62736/diff/1/


Testing
---


Thanks,

Aihua Xu

[jira] [Created] (HIVE-17624) MapredLocakTask running in separate JVM could throw ClassNotFoundException

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17624:
---

 Summary: MapredLocakTask running in separate JVM could throw 
ClassNotFoundException 
 Key: HIVE-17624
 URL: https://issues.apache.org/jira/browse/HIVE-17624
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


{noformat}
set hive.auto.convert.join=true;
set hive.auto.convert.join.use.nonstaged=false;

add jar hive-hcatalog-core.jar;

drop table if exists t1;
CREATE TABLE t1 (a string, b string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';

LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
select * from t1 l join t1 r on l.a=r.a;
{noformat}

The join will use a MapJoin which uses MapredLocalTask in a separate JVM to 
load the table into a Hashmap. But hive doesn't pass added jar to the classpath 
in such JVM so the following exception is thrown.

{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception 
java.lang.ClassNotFoundException: 
org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException: 
java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
at 
org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
at 
org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hive.hcatalog.data.JsonSerDe
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at 
org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
... 15 more

at 
org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17619) Exclude avatica-core.jar since avatica.jar is included

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17619:
---

 Summary: Exclude avatica-core.jar since avatica.jar is included
 Key: HIVE-17619
 URL: https://issues.apache.org/jira/browse/HIVE-17619
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


avatica.jar is included in the project but this jar has a dependency on 
avatica-core.jar and it's pulled into the project as well. 

If avatica-core.jar is included in the classpath in front of  avatica.jar, then 
hive could run into missing class which is shaded inside avatica.jar.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17583) Fix test failure TestAccumuloCliDriver caused from the accumulo version upgrade

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17583:
---

 Summary: Fix test failure TestAccumuloCliDriver caused from the 
accumulo version upgrade
 Key: HIVE-17583
 URL: https://issues.apache.org/jira/browse/HIVE-17583
 Project: Hive
  Issue Type: Test
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17376) Upgrade snappy version to 1.1.4

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17376:
---

 Summary: Upgrade snappy version to 1.1.4
 Key: HIVE-17376
 URL: https://issues.apache.org/jira/browse/HIVE-17376
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Upgrade the snappy java version to 1.1.4. The older version has some issues 
like memory leak (https://github.com/xerial/snappy-java/issues/91).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17373) Upgrade some dependency versions

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17373:
---

 Summary: Upgrade some dependency versions
 Key: HIVE-17373
 URL: https://issues.apache.org/jira/browse/HIVE-17373
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Upgrade some libraries including log4j to 2.8.2, accumulo to 1.8.1 and 
commons-httpclient to 3.1. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17357) Similar to HIVE-17336, plugin jars are not properly added

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17357:
---

 Summary: Similar to HIVE-17336, plugin jars are not properly added
 Key: HIVE-17357
 URL: https://issues.apache.org/jira/browse/HIVE-17357
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


I forgot to include the same change for LocalHiveSparkClient.java in 
HIVE-17336. We need to make the same change as HIVE-17336 in 
LocalHiveSparkClient class to include plugin jars. Maybe we should have a 
common base class for both LocalHiveSparkClient and RemoteHiveSparkClient to 
have some common functions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17353:
---

 Summary: The ResultSets are not accessible if running multiple 
queries within the same HiveStatement 
 Key: HIVE-17353
 URL: https://issues.apache.org/jira/browse/HIVE-17353
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The following queries would fail,
{noformat}
ResultSet rs1 =
stmt.executeQuery("select * from testMultipleResultSets1");
ResultSet rs2 =
stmt.executeQuery("select * from testMultipleResultSets2");
rs1.next();
rs2.next();
{noformat}

with the exception:
{noformat}
[HiveServer2-Handler-Pool: Thread-208]: Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle: 
OperationHandle [opType=EXECUTE_STATEMENT, 
getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
at 
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17336) Missing class 'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from Hive on Spark when inserting into hbase based table

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17336:
---

 Summary: Missing class 
'org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat' from Hive on Spark 
when inserting into hbase based table
 Key: HIVE-17336
 URL: https://issues.apache.org/jira/browse/HIVE-17336
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


When inserting into a hbase based table from hive on spark, the following 
exception is thrown 
{noformat}
Error while processing statement: FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask. 
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: 
org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
Serialization trace:
inputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
 at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
 at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
 at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:183)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:201)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
 at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
 at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
 at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:178)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
 at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
 at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:216)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790

[jira] [Created] (HIVE-17272) when hive.vectorized.execution.enabled is true, query on empty table fails with NPE

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17272:
---

 Summary: when hive.vectorized.execution.enabled is true, query on 
empty table fails with NPE
 Key: HIVE-17272
 URL: https://issues.apache.org/jira/browse/HIVE-17272
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


{noformat}
set hive.vectorized.execution.enabled=true;
CREATE TABLE `tab`(`x` int) PARTITIONED BY ( `y` int);
select * from tab t1 join tab t2 where t1.x=t2.x;
{noformat}

The query fails with the following exception.
{noformat}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.createAndInitPartitionContext(VectorMapOperator.java:386)
 ~[hive-exec-2.3.0.jar:2.3.0]
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.internalSetChildren(VectorMapOperator.java:559)
 ~[hive-exec-2.3.0.jar:2.3.0]
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setChildren(VectorMapOperator.java:474)
 ~[hive-exec-2.3.0.jar:2.3.0]
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106) 
~[hive-exec-2.3.0.jar:2.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_101]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_101]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) 
~[hadoop-common-2.6.0.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) 
~[hadoop-common-2.6.0.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 
~[hadoop-common-2.6.0.jar:?]
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) 
~[hadoop-core-2.6.0-mr1-cdh5.4.2.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_101]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_101]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) 
~[hadoop-common-2.6.0.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) 
~[hadoop-common-2.6.0.jar:?]
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 
~[hadoop-common-2.6.0.jar:?]
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) 
~[hadoop-core-2.6.0-mr1-cdh5.4.2.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
~[hadoop-core-2.6.0-mr1-cdh5.4.2.jar:?]
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268)
 ~[hadoop-core-2.6.0-mr1-cdh5.4.2.jar:?]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[?:1.8.0_101]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[?:1.8.0_101]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181158
---


Ship it!




Ship It!

- Aihua Xu


On July 20, 2017, 10:45 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> ---
> 
> (Updated July 20, 2017, 10:45 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the 
> Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is 
> finished
> it keeps the OutputStream in the Appender open even when the query completes.
> The stream holds a file descriptor and so we leak file descriptors. Note that 
> we
> are already careful to close any streams reading from the operation log file.
> To fix this we use a technique described in the comments of LOG4J2-510 which
> uses reflection to close the appender. The test in TestOperationLoggingLayout 
> is
> extended to check that the Appender is closed.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java 
> e697b545984555414e27bafe92d7f22829a22687 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743 
> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/1/
> 
> 
> Testing
> ---
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender 
> is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

[jira] [Created] (HIVE-17155) findConfFile() in HiveConf.java has some issues with the conf path

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17155:
---

 Summary: findConfFile() in HiveConf.java has some issues with the 
conf path
 Key: HIVE-17155
 URL: https://issues.apache.org/jira/browse/HIVE-17155
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Priority: Minor


In findConfFile() function of HiveConf.java, here are some issues. 
File.pathSeparator which is ":" is used as the separator rather than "/". new 
File(jarUri).getParentFile() will get the "$hive_home/lib" folder, but actually 
we want "$hive_home".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed

[Aihua Xu]

> On July 20, 2017, 11:59 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/cli/operation/Operation.java
> > Lines 269 (patched)
> > <https://reviews.apache.org/r/61010/diff/1/?file=1780484#file1780484line269>
> >
> > We only register the Operation log appender once for all the operation 
> > logs at the beginning. Now if you stop the appender here, then the 
> > following queries will not be able to output to the appender any more, 
> > right?
> > 
> > Can you test your patch by: connect to HS2 from one beeline session, 
> > disconnect and reconnect. Then see if you still see output from the beeline 
> > console?
> > 
> > Will we be able to close OutputStream instead?  stopQueryAppender() 
> > should be called when HS2 service gets shutdown.
> 
> Peter Vary wrote:
> Nice catch Andrew!
> 
> One more thing. Could you please do this for the LogDivertAppenderForTest 
> too? It is only used for tests, but it would be good to clean up it too.
> 
> Thanks,
> Peter
> 
> Andrew Sherman wrote:
> @Aihua thanks for the stimulating question. I ran hand tests to prove 
> that logging works for multiple queries in the same session, and also in a 
> new session. The reason the code is OK is that it is not the RoutineAppender 
> that is closed, but the specific Appender for the query. In 
> https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender 
> this Appender is referred to as a subordinate Appender. I'm updating the code 
> to make this clearer.
> 
> @Peter I will look at LogDivertAppenderForTest to see if I can do the 
> same thing there.

I see. That makes sense. I will take a second look. Thanks for the explanation.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181083
---


On July 20, 2017, 10:45 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> ---
> 
> (Updated July 20, 2017, 10:45 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the 
> Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is 
> finished
> it keeps the OutputStream in the Appender open even when the query completes.
> The stream holds a file descriptor and so we leak file descriptors. Note that 
> we
> are already careful to close any streams reading from the operation log file.
> To fix this we use a technique described in the comments of LOG4J2-510 which
> uses reflection to close the appender. The test in TestOperationLoggingLayout 
> is
> extended to check that the Appender is closed.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java 
> e697b545984555414e27bafe92d7f22829a22687 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743 
> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/1/
> 
> 
> Testing
> ---
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender 
> is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 61010: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181083
---




service/src/java/org/apache/hive/service/cli/operation/Operation.java
Lines 269 (patched)
<https://reviews.apache.org/r/61010/#comment256514>

We only register the Operation log appender once for all the operation logs 
at the beginning. Now if you stop the appender here, then the following queries 
will not be able to output to the appender any more, right?

Can you test your patch by: connect to HS2 from one beeline session, 
disconnect and reconnect. Then see if you still see output from the beeline 
console?

Will we be able to close OutputStream instead?  stopQueryAppender() should 
be called when HS2 service gets shutdown.


- Aihua Xu


On July 20, 2017, 10:45 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> ---
> 
> (Updated July 20, 2017, 10:45 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the 
> Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file.  As log4j does not know when a query is 
> finished
> it keeps the OutputStream in the Appender open even when the query completes.
> The stream holds a file descriptor and so we leak file descriptors. Note that 
> we
> are already careful to close any streams reading from the operation log file.
> To fix this we use a technique described in the comments of LOG4J2-510 which
> uses reflection to close the appender. The test in TestOperationLoggingLayout 
> is
> extended to check that the Appender is closed.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java
>  1a8337f574bb753e8c3c48a6b477b17700b05256 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java 
> e697b545984555414e27bafe92d7f22829a22687 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 8d453d5d9153c2ec86c4adc7a68bd3b5dd249743 
> 
> 
> Diff: https://reviews.apache.org/r/61010/diff/1/
> 
> 
> Testing
> ---
> 
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the Appender 
> is closed.
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

[jira] [Created] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17048:
---

 Summary: Pass HiveOperation info to HiveSemanticAnalyzerHook 
through HiveSemanticAnalyzerHookContext
 Key: HIVE-17048
 URL: https://issues.apache.org/jira/browse/HIVE-17048
 Project: Hive
  Issue Type: Improvement
  Components: Hooks
Affects Versions: 2.1.1
Reporter: Aihua Xu
Assignee: Aihua Xu


Currently hive passes the following info to HiveSemanticAnalyzerHook through 
HiveSemanticAnalyzerHookContext (see 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
 But the operation type (HiveOperation) is also needed in some cases, e.g., 
when integrating with Sentry. 

{noformat}
hookCtx.setConf(conf);
hookCtx.setUserName(userName);
hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
hookCtx.setCommand(command);
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Hadoop3 + hbase2 support in Hive

[Aihua Xu]

Hi hive community,

Hadoop and Hbase released new alpha versions. I'm wondering what's our plan
to support those versions in Hive?

Are we planning to use Hive3 (master) branch to support them and how
typically do we start the upstream work for it?

Thanks
Aihua

[jira] [Created] (HIVE-16911) Upgrade groovy version to 2.4.11

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16911:
---

 Summary: Upgrade groovy version to 2.4.11
 Key: HIVE-16911
 URL: https://issues.apache.org/jira/browse/HIVE-16911
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Hive currently uses groovy 2.4.4 which has security issue 
(https://access.redhat.com/security/cve/cve-2016-6814). Need to upgrade to 
2.4.8 or later. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16902) investigate "failed to remove operation log" errors

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16902:
---

 Summary: investigate "failed to remove operation log" errors
 Key: HIVE-16902
 URL: https://issues.apache.org/jira/browse/HIVE-16902
 Project: Hive
  Issue Type: Bug
  Components: Logging
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


When we call {{set a=3;}} from beeline, the following exception is thrown. 
{noformat}
[HiveServer2-Handler-Pool: Thread-46]: Failed to remove corresponding log file 
of operation: OperationHandle [opType=GET_TABLES, 
getHandleIdentifier()=50f58d7b-f935-4590-922f-de7051a34658]
java.io.FileNotFoundException: File does not exist: 
/var/log/hive/operation_logs/7f613077-e29d-484a-96e1-43c81f9c0999/hive_20170531101400_28d52b7d-ffb9-4815-8c6c-662319628915
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
at 
org.apache.hadoop.hive.ql.session.OperationLog$LogFile.remove(OperationLog.java:122)
at 
org.apache.hadoop.hive.ql.session.OperationLog.close(OperationLog.java:90)
at 
org.apache.hive.service.cli.operation.Operation.cleanupOperationLog(Operation.java:287)
at 
org.apache.hive.service.cli.operation.MetadataOperation.close(MetadataOperation.java:58)
at 
org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationManager.java:273)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.closeOperation(HiveSessionImpl.java:822)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy38.closeOperation(Unknown Source)
at 
org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:475)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:671)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1677)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1662)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:605)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16884) Replace the deprecated HBaseInterface with Table

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16884:
---

 Summary: Replace the deprecated HBaseInterface with Table  
 Key: HIVE-16884
 URL: https://issues.apache.org/jira/browse/HIVE-16884
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HBaseInterface has been deprecated and will get removed in HBase 2.0 by 
HBASE-13395. Replace it with the new one 
{{org.apache.hadoop.hbase.client.Table}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16849) Upgrade jetty version to 9.4.6.v20170531

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16849:
---

 Summary: Upgrade jetty version to 9.4.6.v20170531
 Key: HIVE-16849
 URL: https://issues.apache.org/jira/browse/HIVE-16849
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.0.0
Reporter: Aihua Xu


>From HIVE-16846, the test case of TestJdbcWithMiniHS2#testHttpHeaderSize is 
>returning http error code 413 (PAYLOAD_TOO_LARGE_413) rather than 431 
>(REQUEST_HEADER_FIELDS_TOO_LARGE_431 ) while 431 seems more accurate and the 
>newer version of jetty fixed such issue.

{noformat}
// This should fail with given HTTP response code 413 in error message, 
since header is more
// than the configured the header size
userName = StringUtils.leftPad("*", 2000);
try {
  conn = getConnection(miniHS2.getJdbcURL(testDbName), userName, 
"password");
} catch (Exception e) {
  assertTrue("Header exception thrown", e != null);
  assertTrue(e.getMessage().contains("HTTP Response code: 413"));
} finally {
  if (conn != null) {
conn.close();
  }
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16846) TestJdbcWithMiniHS2#testHttpHeaderSize test case is not testing in HTTP mode

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16846:
---

 Summary: TestJdbcWithMiniHS2#testHttpHeaderSize test case is not 
testing in HTTP mode
 Key: HIVE-16846
 URL: https://issues.apache.org/jira/browse/HIVE-16846
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


TestJdbcWithMiniHS2#testHttpHeaderSize test case actually is testing binary 
mode so the request/response sizes are not checked. 

We need to build MiniHS2 using withHTTPTransport() to start the HTTP mode. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16769) Possible hive service startup due to the existing of /tmp/stderr

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16769:
---

 Summary: Possible hive service startup due to the existing of 
/tmp/stderr
 Key: HIVE-16769
 URL: https://issues.apache.org/jira/browse/HIVE-16769
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HIVE-12497 prints the ignoring errors from hadoop version, hbase mapredcp and 
hadoop jars to /tmp/${USER}/stderr. 

In some cases ${USER} is not set, then the file becomes /tmp/stderr.  If  such 
file preexists with different permission, it will cause the service startup to 
fail.

I just tried the script without outputting to stderr file, I don't see such 
error any more {{"ERROR StatusLogger No log4j2 configuration file found. Using 
default configuration: logging only errors to the console."}}.

I think we can remove such redirect to avoid possible startup failure.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16682) Check if the console message from the hive schema tool needs to print to logging file

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16682:
---

 Summary: Check if the console message from the hive schema tool 
needs to print to logging file
 Key: HIVE-16682
 URL: https://issues.apache.org/jira/browse/HIVE-16682
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Aihua Xu
Priority: Minor


>From HiveSchemaTool, most of the messages are printed to console and some of 
>them are printed to log. Evaluate the console messages if make sense to print 
>to log as well and what would be the best way to print them to avoid 
>duplication in case if LOG is configured to be console. 





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58936: HIVE-16143 : Improve msck repair batching

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58936/#review174896
---


Ship it!




Ship It!

- Aihua Xu


On May 12, 2017, 9:35 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58936/
> ---
> 
> (Updated May 12, 2017, 9:35 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, and Sahil Takiar.
> 
> 
> Bugs: HIVE-16143
> https://issues.apache.org/jira/browse/HIVE-16143
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16143 : Improve msck repair batching
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> d3ea824c21f2fbf98177cb12a18019416f36a3f9 
>   common/src/java/org/apache/hive/common/util/RetryUtilities.java 
> PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestRetryUtilities.java 
> PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/create_like.q 
> 38f384e4c547d3c93d510b89fccfbc2b8e2cba09 
>   itests/hive-blobstore/src/test/results/clientpositive/create_like.q.out 
> 0d362a716291637404a3859fe81068594d82c9e0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2ae1eacb68cef6990ae3f2050af0bed7c8e9843f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> 22542331621ca4ce5277c2f46a4264b7540a4d1e 
>   ql/src/test/queries/clientpositive/msck_repair_1.q 
> ea596cbbd2d4c230f2b5afbe379fc1e8836b6fbd 
>   ql/src/test/queries/clientpositive/msck_repair_2.q 
> d8338211e970ebac68a7471ee0960ccf2d51cba3 
>   ql/src/test/queries/clientpositive/msck_repair_3.q 
> fdefca121a2de361dbd19e7ef34fb220e1733ed2 
>   ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
> e56e97ac36a6544f3e20478fdb0e8fa783a857ef 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> 2e0d9dc423071ebbd9a55606f196cf7752e27b1a 
>   ql/src/test/results/clientpositive/msck_repair_1.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_2.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_3.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
> ba99024163a1f2c59d59e9ed7ea276c154c99d24 
>   ql/src/test/results/clientpositive/repair.q.out 
> c1834640a35500c521a904a115a718c94546df10 
> 
> 
> Diff: https://reviews.apache.org/r/58936/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 58936: HIVE-16143 : Improve msck repair batching

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58936/#review174792
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3357-3361 (patched)
<https://reviews.apache.org/r/58936/#comment248014>

Vihang and Sahil,

Typically what would cause the batch to fail? Is that because the batch 
could be too large? 

Right now, we are hard coding decayingFactor to 2. I have another thought: 
maybe with the retries, we will  calculate such decayingFactor so the last 
retry will always process one partition at a time just like what we are doing. 
So given batch size 100 and retries 4, 100, 66, 33, 1? 

How do you think?


- Aihua Xu


On May 11, 2017, 12:04 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58936/
> ---
> 
> (Updated May 11, 2017, 12:04 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, and Sahil Takiar.
> 
> 
> Bugs: HIVE-16143
> https://issues.apache.org/jira/browse/HIVE-16143
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16143 : Improve msck repair batching
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> d3ea824c21f2fbf98177cb12a18019416f36a3f9 
>   common/src/java/org/apache/hive/common/util/RetryUtilities.java 
> PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestRetryUtilities.java 
> PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/create_like.q 
> 38f384e4c547d3c93d510b89fccfbc2b8e2cba09 
>   itests/hive-blobstore/src/test/results/clientpositive/create_like.q.out 
> 0d362a716291637404a3859fe81068594d82c9e0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2ae1eacb68cef6990ae3f2050af0bed7c8e9843f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> 22542331621ca4ce5277c2f46a4264b7540a4d1e 
>   ql/src/test/queries/clientpositive/msck_repair_1.q 
> ea596cbbd2d4c230f2b5afbe379fc1e8836b6fbd 
>   ql/src/test/queries/clientpositive/msck_repair_2.q 
> d8338211e970ebac68a7471ee0960ccf2d51cba3 
>   ql/src/test/queries/clientpositive/msck_repair_3.q 
> fdefca121a2de361dbd19e7ef34fb220e1733ed2 
>   ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
> e56e97ac36a6544f3e20478fdb0e8fa783a857ef 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> 2e0d9dc423071ebbd9a55606f196cf7752e27b1a 
>   ql/src/test/results/clientpositive/msck_repair_1.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_2.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_3.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
> ba99024163a1f2c59d59e9ed7ea276c154c99d24 
>   ql/src/test/results/clientpositive/repair.q.out 
> c1834640a35500c521a904a115a718c94546df10 
> 
> 
> Diff: https://reviews.apache.org/r/58936/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

[jira] [Created] (HIVE-16647) Improve the validation output to make the output to stderr and stdout more consistent

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16647:
---

 Summary: Improve the validation output to make the output to 
stderr and stdout more consistent
 Key: HIVE-16647
 URL: https://issues.apache.org/jira/browse/HIVE-16647
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 2.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Priority: Minor


Some output are printed to stderr or stdout inconsistently. Here are some of 
them. Update to make them more consistent.

*  Version table validation
  When the version table is missing, the err msg goes to stderr
  When the version table is not valid, the err msg goes to stdout with a 
message like "Failed in schema version validation: 
*  Metastore/schema table validation
** When the version table contains the wrong version or there are no rows in 
the version table, err msg goes to stderr
** When there diffs between the schema and metastore tables, the err msg goes 
to stdout




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59096: HIVE-16607 ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59096/#review174465
---


Ship it!




Ship It!

- Aihua Xu


On May 10, 2017, 9:17 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59096/
> ---
> 
> (Updated May 10, 2017, 9:17 a.m.)
> 
> 
> Review request for hive, Aihua Xu and pengcheng xiong.
> 
> 
> Bugs: HIVE-16607
> https://issues.apache.org/jira/browse/HIVE-16607
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When creating a QueryState object the caller could specify if new QueryID 
> should be created or the exisiting should be used.
> Created a QueryStateBuilder to make the QueryState object creation more 
> readable.
> New QueryId is only created in two places:
> - Driver constructor
> - Operation constructor
> Otherwise the existing queryId is used
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
>  6ff48ee 
>   itests/src/test/resources/testconfiguration.properties 5ab3076 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java 3d9ca99 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 7c50e18 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29cce9a 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 6dfaa9f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cf575de 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 
> 77bce97 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  b121eea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> 3b719af 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java c7266bc 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> c734988 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBCompact.java 201622e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBJoinTreeApplyPredicate.java 
> e607f10 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBSubQuery.java 2674835 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java
>  80865bd 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSemanticAnalyzerFactory.java 
> 5849950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
>  a573808 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  58cb4b4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV1.java
>  5d01080 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV2.java
>  c552ba7 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0b27608 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 0b51591 
> 
> 
> Diff: https://reviews.apache.org/r/59096/diff/3/
> 
> 
> Testing
> ---
> 
> Added new BeeLine test - The original code made the test output different 
> from the Cli test output, since the QueryLog was truncated when the queryId 
> was changed. After the change the BeeLine test output is exactly the same as 
> the Cli output.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58936: HIVE-16143 : Improve msck repair batching

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58936/#review174367
---




common/src/java/org/apache/hive/common/util/RetryUtilities.java
Lines 78-82 (patched)
<https://reviews.apache.org/r/58936/#comment247490>

That seems to be a good idea.


- Aihua Xu


On May 2, 2017, 10:25 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58936/
> ---
> 
> (Updated May 2, 2017, 10:25 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Sahil Takiar.
> 
> 
> Bugs: HIVE-16143
> https://issues.apache.org/jira/browse/HIVE-16143
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16143 : Improve msck repair batching
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/RetryUtilities.java 
> PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestRetryUtilities.java 
> PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/create_like.q 
> 38f384e4c547d3c93d510b89fccfbc2b8e2cba09 
>   itests/hive-blobstore/src/test/results/clientpositive/create_like.q.out 
> 0d362a716291637404a3859fe81068594d82c9e0 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2ae1eacb68cef6990ae3f2050af0bed7c8e9843f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 917e565f28b2c9aaea18033ea3b6b20fa41fcd0a 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_0.q 
> 22542331621ca4ce5277c2f46a4264b7540a4d1e 
>   ql/src/test/queries/clientpositive/msck_repair_1.q 
> ea596cbbd2d4c230f2b5afbe379fc1e8836b6fbd 
>   ql/src/test/queries/clientpositive/msck_repair_2.q 
> d8338211e970ebac68a7471ee0960ccf2d51cba3 
>   ql/src/test/queries/clientpositive/msck_repair_3.q 
> fdefca121a2de361dbd19e7ef34fb220e1733ed2 
>   ql/src/test/queries/clientpositive/msck_repair_batchsize.q 
> e56e97ac36a6544f3e20478fdb0e8fa783a857ef 
>   ql/src/test/results/clientpositive/msck_repair_0.q.out 
> 2e0d9dc423071ebbd9a55606f196cf7752e27b1a 
>   ql/src/test/results/clientpositive/msck_repair_1.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_2.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_3.q.out 
> 3f2fe75b194f1248bd5c073dd7db6b71b2ffc2ba 
>   ql/src/test/results/clientpositive/msck_repair_batchsize.q.out 
> ba99024163a1f2c59d59e9ed7ea276c154c99d24 
>   ql/src/test/results/clientpositive/repair.q.out 
> c1834640a35500c521a904a115a718c94546df10 
> 
> 
> Diff: https://reviews.apache.org/r/58936/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 59096: HIVE-16607 ColumnStatsAutoGatherContext regenerates HiveConf.HIVEQUERYID

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59096/#review174322
---




ql/src/java/org/apache/hadoop/hive/ql/QueryState.java
Lines 79 (patched)
<https://reviews.apache.org/r/59096/#comment247456>

I suggest to rename it to just Builder and use this class as 
QueryState.Builder.



ql/src/java/org/apache/hadoop/hive/ql/QueryState.java
Lines 96 (patched)
<https://reviews.apache.org/r/59096/#comment247465>

Maybe name those functions like "withRunAsync", "withConfOverlay", 
"withGenerateNewQueryId"?

And also add a new withConf() since conf can be optional as well. So 
build() will not accept any parameters.


- Aihua Xu


On May 9, 2017, 3:05 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59096/
> ---
> 
> (Updated May 9, 2017, 3:05 p.m.)
> 
> 
> Review request for hive, Aihua Xu and pengcheng xiong.
> 
> 
> Bugs: HIVE-16607
> https://issues.apache.org/jira/browse/HIVE-16607
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When creating a QueryState object the caller could specify if new QueryID 
> should be created or the exisiting should be used.
> Created a QueryStateBuilder to make the QueryState object creation more 
> readable.
> New QueryId is only created in two places:
> - Driver constructor
> - Operation constructor
> Otherwise the existing queryId is used
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
>  6ff48ee 
>   itests/src/test/resources/testconfiguration.properties 5ab3076 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java 3d9ca99 
>   itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 
> 7c50e18 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29cce9a 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 6dfaa9f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 
> 77bce97 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  b121eea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> 3b719af 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java c7266bc 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> c734988 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBCompact.java 201622e 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBJoinTreeApplyPredicate.java 
> e607f10 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBSubQuery.java 2674835 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java
>  80865bd 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSemanticAnalyzerFactory.java 
> 5849950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java
>  a573808 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  58cb4b4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV1.java
>  5d01080 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV2.java
>  c552ba7 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 0b27608 
> 
> 
> Diff: https://reviews.apache.org/r/59096/diff/1/
> 
> 
> Testing
> ---
> 
> Added new BeeLine test - The original code made the test output different 
> from the Cli test output, since the QueryLog was truncated when the queryId 
> was changed. After the change the BeeLine test output is exactly the same as 
> the Cli output.
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 59020: Support Parquet through HCatalog

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59020/#review174039
---




hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileRecordWriterContainer.java
Lines 30 (patched)
<https://reviews.apache.org/r/59020/#comment247146>

This change looks good to me. 

Sergio, can you also help review the patch since you are more faimilar with 
Parquet?


- Aihua Xu


On May 5, 2017, 8:11 a.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59020/
> ---
> 
> (Updated May 5, 2017, 8:11 a.m.)
> 
> 
> Review request for hive, Aihua Xu and Sergio Pena.
> 
> 
> Bugs: HIVE-8838
> https://issues.apache.org/jira/browse/HIVE-8838
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding support for HCatalog to write tables stored in Parquet format
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileRecordWriterContainer.java
>  b2abc5fbb3670893415354552239d67d072459ed 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/SpecialCases.java
>  60af5c0bf397273fb820f0ee31e578745dbc200f 
>   
> hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatLoaderComplexSchema.java
>  4c686fec596d39d41d458bc3ea2753877bd9df98 
>   
> hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatLoaderEncryption.java
>  ad11eab1b7e67541b56e90e4a85ba37b41a4db92 
>   
> hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatStorerMulti.java
>  918332ddfda58306707d326f8668b2c223110a29 
>   
> hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestParquetHCatLoader.java
>  6cd382145b55d6b85fc3366faeaba2aaef65ab04 
>   
> hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestParquetHCatStorer.java
>  6dfdc04954dd0b110b1a7194e69468b5dc2f842e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  a7bb5eedbb99f3cea4601b9fce9a0ad3461567d0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java 
> b339cc4347eea143dca2f6d98f9aaafdc427 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java
>  71a78cf040667bf14b6c720373e4acd102da19f4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
>  c021dafa480e65d7c0c19a5a85988464112468cb 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestMapredParquetOutputFormat.java
>  ec85b5df0f95cbd45b87259346ae9c1e5aa604a4 
> 
> 
> Diff: https://reviews.apache.org/r/59020/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on cluster, and re-enabled previously disabled tests in HCatalog (for 
> Parquet) that were failing (this adds ~40 tests to be run)
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

[jira] [Created] (HIVE-16528) Exclude older version of beanutils from dependent jars in test pom.xml

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16528:
---

 Summary: Exclude older version of beanutils from dependent jars in 
test pom.xml 
 Key: HIVE-16528
 URL: https://issues.apache.org/jira/browse/HIVE-16528
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The test build is picking up the older beanutils jars. That is causing test 
failures when hadoop is upgrading to alpha2.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58456: Query cancel: improve the way to handle files

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58456/#review172209
---


Ship it!




Ship It!

- Aihua Xu


On April 14, 2017, 1:14 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58456/
> ---
> 
> (Updated April 14, 2017, 1:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, and Sergio Pena.
> 
> 
> Bugs: HIVE-16426
> https://issues.apache.org/jira/browse/HIVE-16426
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Use threadlocal variable to store cancel state to make it is accessible 
> without being passed around by parameters. 
> 2. Add checkpoints for file operations.
> 3. Remove backgroundHandle.cancel to avoid failed file cleanup because of the 
> interruption. By what I observed that the method seems not very effective for 
> scheduled operation, for example, the on going HMS API calls.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> a80004662068eb2391c0dd7062f77156b222375b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> b0657f01d4482dc8bb8dc180e5e7deffbdb533e6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
> 7a113bf8e5c4dd8c2c486741a5ebc7b8940e746b 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04fc0a17c93120b8f6e6d7c36e4d70631d56baca 
> 
> 
> Diff: https://reviews.apache.org/r/58456/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

Re: Review Request 58456: Query cancel: improve the way to handle files

[Aihua Xu]

> On April 14, 2017, 5:43 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
> > Lines 399 (patched)
> > <https://reviews.apache.org/r/58456/diff/1/?file=1692689#file1692689line399>
> >
> > I'm not exactly following what we are doing here. Not sure how 
> > background thread gets closed later.
> > 
> > Otherwise, the other changes look good.
> 
> Yongzhi Chen wrote:
> The background thread will complete the task or gracefully closed with 
> the guidance of the cancel status. Our current cancel design majorly follows 
> the pattern that cancel command set the cancel status, the working 
> thread(background thread) check the cancel status and decide to quit or 
> continue. The backgroundHandle.cancel(true) does not follow the pattern and 
> cause some conflicts. The following warning log is caused by this:
> 2017-04-11 09:57:30,727 WARN  org.apache.hadoop.hive.ql.exec.Utilities: 
> [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted

I see. Got it.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58456/#review172009
---


On April 14, 2017, 1:14 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58456/
> -------
> 
> (Updated April 14, 2017, 1:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, and Sergio Pena.
> 
> 
> Bugs: HIVE-16426
> https://issues.apache.org/jira/browse/HIVE-16426
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Use threadlocal variable to store cancel state to make it is accessible 
> without being passed around by parameters. 
> 2. Add checkpoints for file operations.
> 3. Remove backgroundHandle.cancel to avoid failed file cleanup because of the 
> interruption. By what I observed that the method seems not very effective for 
> scheduled operation, for example, the on going HMS API calls.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> a80004662068eb2391c0dd7062f77156b222375b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> b0657f01d4482dc8bb8dc180e5e7deffbdb533e6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
> 7a113bf8e5c4dd8c2c486741a5ebc7b8940e746b 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04fc0a17c93120b8f6e6d7c36e4d70631d56baca 
> 
> 
> Diff: https://reviews.apache.org/r/58456/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

[jira] [Created] (HIVE-16455) ADD JAR command leaks JAR Files

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16455:
---

 Summary: ADD JAR command leaks JAR Files
 Key: HIVE-16455
 URL: https://issues.apache.org/jira/browse/HIVE-16455
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Aihua Xu
Assignee: Aihua Xu


HiveServer2 is leaking file handles when using ADD JAR statement and the JAR 
file added is not used in the query itself.

{noformat}
beeline> !connect jdbc:hive2://localhost:1 admin
0: jdbc:hive2://localhost:1> create table test_leak (a int);
0: jdbc:hive2://localhost:1> insert into test_leak Values (1);

-- Exit beeline terminal; Find PID of HiveServer2

[root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
0
[root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:1/default -n 
admin

And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from 
test_leak"
[root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
1

java29588 hive  391u   REG  252,3125987  2099944 
/tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
java29588 hive  392u   REG  252,3125987  2099946 
/tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
java29588 hive  393r   REG  252,3125987  2099825 
/tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
java29588 hive  394r   REG  252,3125987  2099833 
/tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
java29588 hive  395r   REG  252,3125987  2099827 
/tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
java29588 hive  396r   REG  252,3125987  2099822 
/tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
java29588 hive  397r   REG  252,3125987  2099831 
/tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
java29588 hive  399r   REG  252,3125987  2099835 
/tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)
{noformat}
You can see the the session directory (and therefore anything in it) is set to 
delete only on exit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58456: Query cancel: improve the way to handle files

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58456/#review172009
---




service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
Lines 399 (patched)
<https://reviews.apache.org/r/58456/#comment245060>

I'm not exactly following what we are doing here. Not sure how background 
thread gets closed later.

Otherwise, the other changes look good.


- Aihua Xu


On April 14, 2017, 1:14 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58456/
> ---
> 
> (Updated April 14, 2017, 1:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, and Sergio Pena.
> 
> 
> Bugs: HIVE-16426
> https://issues.apache.org/jira/browse/HIVE-16426
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Use threadlocal variable to store cancel state to make it is accessible 
> without being passed around by parameters. 
> 2. Add checkpoints for file operations.
> 3. Remove backgroundHandle.cancel to avoid failed file cleanup because of the 
> interruption. By what I observed that the method seems not very effective for 
> scheduled operation, for example, the on going HMS API calls.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> a80004662068eb2391c0dd7062f77156b222375b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> b0657f01d4482dc8bb8dc180e5e7deffbdb533e6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
> 7a113bf8e5c4dd8c2c486741a5ebc7b8940e746b 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 04fc0a17c93120b8f6e6d7c36e4d70631d56baca 
> 
> 
> Diff: https://reviews.apache.org/r/58456/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

[jira] [Created] (HIVE-16450) Some metastore operations are not retried even with desired underlining exceptions

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16450:
---

 Summary: Some metastore operations are not retried even with 
desired underlining exceptions
 Key: HIVE-16450
 URL: https://issues.apache.org/jira/browse/HIVE-16450
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


In RetryingHMSHandler class, we are expecting the operations should retry when 
the cause of MetaException is JDOException or NucleusException.
{noformat}
if (e.getCause() instanceof MetaException && e.getCause().getCause() != 
null) {
  if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
  e.getCause().getCause() instanceof NucleusException) {
// The JDOException or the Nucleus Exception may be wrapped further 
in a MetaException
caughtException = e.getCause().getCause();
   }
{noformat}

While in ObjectStore, many places we are only throwing new MetaException(msg) 
without the cause, so we are missing retrying for some cases. e.g., with the 
following JDOException, we should retry but it's ignored.

{noformat}
2017-04-04 17:28:21,602 ERROR metastore.ObjectStore 
(ObjectStore.java:getMTableColumnStatistics(6555)) - Error retrieving 
statistics via jdo
javax.jdo.JDOException: Exception thrown when executing query
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:6546)
at 
org.apache.hadoop.hive.metastore.ObjectStore.access$1200(ObjectStore.java:171)
at 
org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6606)
at 
org.apache.hadoop.hive.metastore.ObjectStore$9.getJdoResult(ObjectStore.java:6595)
at 
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2633)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6594)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6588)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
at com.sun.proxy.$Proxy0.getTableColumnStatistics(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTableUpdateTableColumnStats(HiveAlterHandler.java:787)
at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:247)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3809)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3779)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy3.alter_table_with_environment_context(Unknown 
Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9617)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:9601)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{

[jira] [Created] (HIVE-16439) Exclude older v2 version of jackson lib from pom.xml

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16439:
---

 Summary: Exclude older v2 version of jackson lib from pom.xml 
 Key: HIVE-16439
 URL: https://issues.apache.org/jira/browse/HIVE-16439
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Aihua Xu
Assignee: Aihua Xu


There are multiple versions of jackson libs included in the dependent jars like 
spark-client and metrics-json. That causes older versions of jackson libs to be 
used.   

We need to exclude them from the dependencies and use the explicit one 
(currently 2.6.5).

  com.fasterxml.jackson.core
  jackson-databind




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)