[jira] [Created] (HIVE-22301) Hive lineage is not generated for insert overwrite queries on partitioned tables

2019-10-07 Thread Sidharth Kumar Mishra (Jira) 
Sidharth Kumar Mishra created HIVE-22301:


 Summary: Hive lineage is not generated for insert overwrite 
queries on partitioned tables
 Key: HIVE-22301
 URL: https://issues.apache.org/jira/browse/HIVE-22301
 Project: Hive
  Issue Type: Bug
Reporter: Sidharth Kumar Mishra
 Attachments: ScreenShot HookContext.png, ScreenShot 
RunPostExecHook.png, ScreenShot runBeforeExecution.png

Problem: When I run the below mentioned queries, the last query should have 
given the proper hive lineage info (through HookContext) from table_b to 
table_t.
 * Create table table_t (id int) partitioned by (dob date);
 * Create table table_b (id int) partitioned by (dob date);
 * from table_b a insert overwrite table table_t select a.id,a.dob;

Note : for CTAS query from a partitioned table , this issue is not seen. Only 
for insert queries like insert into  select * from  and query 
like above, issue is seen.

This issue is seen in latest HDP builds as well.

 

Technical Observations:

At HookContext (passed from hive.ql.Driver to Hive Hook of Atlas through 
hookRunner.runPostExecHooks call) contains no outputs. Check below screenshot 
from IntelliJ.

!ScreenShot RunPostExecHook.png!

 

I found that the PrivateHookContext is getting created with proper outputs 
value as shown below initially:

  !ScreenShot HookContext.png!

The same is passed properly to runBeforeExecutionHook as shown below:

!ScreenShot runBeforeExecution.png!

 

Later when we pass HookContext to runPostExecHooks, there is no output 
populated. Kindly check the reason and let me know if you need any further 
information from my end.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22300) Deduplicate the authentication and LDAP code in HMS and HS2

2019-10-07 Thread Ashutosh Bapat (Jira) 
Ashutosh Bapat created HIVE-22300:
-

 Summary: Deduplicate the authentication and LDAP code in HMS and 
HS2
 Key: HIVE-22300
 URL: https://issues.apache.org/jira/browse/HIVE-22300
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2, Standalone Metastore
Reporter: Ashutosh Bapat
Assignee: Ashutosh Bapat


HIVE-22267 has duplicated code from hive-service/auth directory under 
standalone-metastore directory. Deduplicate this code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71575: HIVE-22284: Improve LLAP CacheContentsTracker to collect and display correct statistics

2019-10-07 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71575/
---

(Updated Oct. 7, 2019, 2:26 p.m.)


Review request for hive.


Changes
---

Reworked CacheTag to minimize memory overhead footprint


Bugs: HIVE-22284
https://issues.apache.org/jira/browse/HIVE-22284


Repository: hive-git


Description
---

When keeping track of which buffers correspond to what Hive objects, 
CacheContentsTracker relies on cache tags.

Currently a tag is a simple String that ideally holds DB and table name, and a 
partition spec concatenated by . and / . The information here is derived from 
the Path of the file that is getting cached. Needless to say sometimes this 
produces a wrong tag especially for external tables.

Also there's a bug when calculating aggregated stats for a 'parent' tag 
(corresponding to the table of the partition) because the overall maxCount and 
maxSize do not add up to the sum of those in the partitions. This happens when 
buffers get removed from the cache.


Diffs (updated)
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 
a351a193c6bc558bb420049c54b7657cd7d04b7c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 64c0125833af100fd7012b9751d075ab536ad1b0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java 
f91a5d91a5b739dcbee98a1485ad4c59f6a9057b 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapDataBuffer.java 
405fca2d4fae9fe0e3fd6d6d1345d55255d6df78 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
4dd3826a67dfff66ce9c90027d61a9012c0a15e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
62d7e5534486b53634de332875c5fd5d336c29b4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 2a39d2d32807a51346baad28b04d87670381b6d5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
41855e171eaa5bf8da638bc62bce3d0d49dc4bae 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
c63ee5f79b4f9fc356f033960e0af1a7b0058038 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 1378a01f44ef774a15f769460833064c6305b2d6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 2a0c5ca92f3c7431f3c399f309a538f47eb27597 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 85a42f945624c3ca468790772f52363b4064d8fc 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 d414b1405b7672767196b3eaad02baa516169288 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
8400fe98411ed07bd525a51a223fc35423136efb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 30dc1b9da2002689b8b1917f46ae3ca24194f3be 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestCacheContentsTracker.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java 
af04a51b5536550b2d2f7d3e008cf2b2dea607d4 
  ql/src/java/org/apache/hadoop/hive/llap/LlapHiveUtils.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
241a3001e6e0002377736d6d0e820fde004b0bac 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 
210c987b7f580dacda5bdb487af9cf234a738b79 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a9a9f101948a970e0dbf2f77eeb6f688a88d1cbd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 61e2556b08fe4247f35673f24378505ada20a605 
  storage-api/src/java/org/apache/hadoop/hive/common/io/CacheTag.java 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 
2ac0a18a5026e76e65c3c3a8b81d5a844c472ed2 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
d7de3619380d24a1aeea2bac9a66485d7d468517 


Diff: https://reviews.apache.org/r/71575/diff/3/

Changes: https://reviews.apache.org/r/71575/diff/2-3/


Testing
---


Thanks,

Adam Szita

[jira] [Created] (HIVE-22299) Use dependencyManagement in pom.xml-s

2019-10-07 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-22299:
---

 Summary: Use dependencyManagement in pom.xml-s
 Key: HIVE-22299
 URL: https://issues.apache.org/jira/browse/HIVE-22299
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


The biggest advantage would be to have command line property overrides working 
reliably

related: HIVE-14289 MNG-6237



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta

2019-10-07 Thread Peter Vary (Jira) 
Peter Vary created HIVE-22298:
-

 Summary: Allow Llap IO cache for reading tables without delete 
delta
 Key: HIVE-22298
 URL: https://issues.apache.org/jira/browse/HIVE-22298
 Project: Hive
  Issue Type: Improvement
Reporter: Peter Vary
Assignee: Peter Vary


Since HIVE-19985 we can use Llap IO cache for queries which are not explicitly 
reading ROW_ID even for original files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22297) qtests: add regex based replacer

2019-10-07 Thread Zoltan Haindrich (Jira) 
Zoltan Haindrich created HIVE-22297:
---

 Summary: qtests: add regex based replacer
 Key: HIVE-22297
 URL: https://issues.apache.org/jira/browse/HIVE-22297
 Project: Hive
  Issue Type: Improvement
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


right now we have some hard wired replacers for specific patterns
idea would be to generalize it - and make it able to configure replacements 
from the qfile



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22296) Set NULLS LAST as the default null ordering in Sorted by clause

2019-10-07 Thread Krisztian Kasa (Jira) 
Krisztian Kasa created HIVE-22296:
-

 Summary: Set NULLS LAST as the default null ordering in Sorted by 
clause
 Key: HIVE-22296
 URL: https://issues.apache.org/jira/browse/HIVE-22296
 Project: Hive
  Issue Type: Improvement
Reporter: Krisztian Kasa


1. In HiveParser.g use nullsLast() function to determine default null ordering. 
This was reverted by (HIVE-22281)
2. store null ordering in metastore



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22295) Javadoc for UDAFs

2019-10-07 Thread Krisztian Kasa (Jira) 
Krisztian Kasa created HIVE-22295:
-

 Summary: Javadoc for UDAFs
 Key: HIVE-22295
 URL: https://issues.apache.org/jira/browse/HIVE-22295
 Project: Hive
  Issue Type: Task
  Components: UDF
Reporter: Krisztian Kasa


{code}
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCumeDist.java:33:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFDenseRank.java:24:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFFirstValue.java:44:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLag.java:35:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLastValue.java:40:@Description(name
 = "last_value", value = "_FUNC_(x)"): warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLead.java:31:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java:41:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentRank.java:34:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRank.java:41:@Description(:
 warning: Missing a Javadoc comment.
./ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRowNumber.java:40:@Description(:
 warning: Missing a Javadoc comment.
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 71586: HIVE-22230: Add support for filtering partitions on temporary tables.

2019-10-07 Thread Laszlo Pinter via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71586/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22230: Add support for filtering partitions on temporary tables.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
00b4301f54a670b638e98af75b32a5e5639559d9 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
db3d9dbca899e5563355ee8ea43b4020afcc71e6 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
e659b5407d212303ba95b7c2f65baab60989be5e 
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java 
c861107ecb1338eafa5c5c84cf36303745676d4e 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientGetPartitionsTempTable.java
 d35621a9c22f2a59653c705b093f19b0c9dc4ff8 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientListPartitionsTempTable.java
 a74b543faa4eb8714be71a9eac6a8f55eaa97dba 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestListPartitions.java
 94d37c220f1ddf510526cb5986df46e5b88e6cc9 


Diff: https://reviews.apache.org/r/71586/diff/1/


Testing
---


Thanks,

Laszlo Pinter