Re: Review Request 71561: HIVE-22250

2019-10-02 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71561/#review218039
---




ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
Lines 171 (patched)


This seems to be rather strict. I think we may call this method when a user 
includes its own UDF? Mainly, I am concerned with custom UDFs that will stop 
working if they do not include a description. Probably a WARN log message is 
more adequate.


- Jesús Camacho Rodríguez


On Oct. 2, 2019, 2:52 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71561/
> ---
> 
> (Updated Oct. 2, 2019, 2:52 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and 
> Vineet Garg.
> 
> 
> Bugs: HIVE-22250
> https://issues.apache.org/jira/browse/HIVE-22250
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Describe function does not provide description for rank functions
> =
> The `DESCRIBE FUNCTION` command gets the description of a function from the 
> `@Description` annotations `value` field. If an UDF is annotated with the 
> `@WindowFunctionDescription` hive prints 
> ```
> There is no documentation for function 
> ```
> Even if the description is present in the `@WindowFunctionDescription` 
> annotation.
> 
> This patch moves the `@WindowFunctionDescription.Description` field to a 
> separate annotation and provide the `@Description` annotation if both 
> annotations are missing.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java fc2a0e1970 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionDescription.java 
> 511d9641c3 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java a7f4bf1fcc 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 8f4ec3b1ef 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 7a0145243d 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 451b45fbbc 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 4fe9c323cc 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java b31eeb08a0 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 315789c1c1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 
> a8bcc972bb 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCumeDist.java 
> 70541fe565 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFDenseRank.java 
> 30bfd2bb8c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFFirstValue.java 
> b8b7d8e6da 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLag.java 
> e0edbb42af 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLastValue.java 
> dadec3b793 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLead.java 
> e678278b8b 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java 
> 8b2812d5bc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentRank.java 
> 1a7c94431b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileCont.java
>  e7e4fda6ea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileDisc.java
>  d7c295cb11 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRank.java 
> a28def73a1 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRowNumber.java 
> 41a3e582ec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
>  c5c73835af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBucketNumber.java 
> 472cc85047 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEpochMilli.java 
> d8e822ae97 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java
>  733fe63e80 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSurrogateKey.java 
> 1372b60724 
>   ql/src/test/queries/clientpositive/desc_function.q d055d9ca03 
>   ql/src/test/results/clientpositive/desc_function.q.out 1f804bba60 
> 
> 
> Diff: https://reviews.apache.org/r/71561/diff/2/
> 
> 
> Testing
> ---
> 
> Added test cases to `desc_function.q`:
> ```
> DESCRIBE FUNCTION dense_rank;
> DESCRIBE FUNCTION EXTENDED dense_rank;
> ```
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

[jira] [Created] (HIVE-22288) Add tests for grouping sets for correlated subqueries

2019-10-02 Thread Steve Carlin (Jira) 
Steve Carlin created HIVE-22288:
---

 Summary: Add tests for grouping sets for correlated subqueries
 Key: HIVE-22288
 URL: https://issues.apache.org/jira/browse/HIVE-22288
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.1.2
Reporter: Steve Carlin
Assignee: Vineet Garg


After the upgrade to Calcite 1.21, the CALCITE_461_FIXED patch was removed.   
The related Jira is CALCITE-574.  

Grouping sets for correlated queries should now work and we need to add tests 
for them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22287) Metastore logging needs to add the partition info

2019-10-02 Thread Gopal Vijayaraghavan (Jira) 
Gopal Vijayaraghavan created HIVE-22287:
---

 Summary: Metastore logging needs to add the partition info
 Key: HIVE-22287
 URL: https://issues.apache.org/jira/browse/HIVE-22287
 Project: Hive
  Issue Type: Bug
  Components: Standalone Metastore
Affects Versions: 4.0.0
Reporter: Gopal Vijayaraghavan


{code}
[mdc@18060 class="utils.MetaStoreUtils" level="WARN" thread="HMSHandler #4"] 
Updating partition stats fast for: web_returns
[mdc@18060 class="utils.MetaStoreUtils" level="WARN" thread="HMSHandler #9"] 
Updated size to 21890 
{code}

The HMS logs are at WARN level, but are missing the partition name in both log 
lines.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 71575: HIVE-22284: Improve LLAP CacheContentsTracker to collect and display correct statistics

2019-10-02 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71575/
---

Review request for hive.


Bugs: HIVE-22284
https://issues.apache.org/jira/browse/HIVE-22284


Repository: hive-git


Description
---

When keeping track of which buffers correspond to what Hive objects, 
CacheContentsTracker relies on cache tags.

Currently a tag is a simple String that ideally holds DB and table name, and a 
partition spec concatenated by . and / . The information here is derived from 
the Path of the file that is getting cached. Needless to say sometimes this 
produces a wrong tag especially for external tables.

Also there's a bug when calculating aggregated stats for a 'parent' tag 
(corresponding to the table of the partition) because the overall maxCount and 
maxSize do not add up to the sum of those in the partitions. This happens when 
buffers get removed from the cache.


Diffs
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 
a351a193c6bc558bb420049c54b7657cd7d04b7c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 64c0125833af100fd7012b9751d075ab536ad1b0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java 
f91a5d91a5b739dcbee98a1485ad4c59f6a9057b 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapDataBuffer.java 
405fca2d4fae9fe0e3fd6d6d1345d55255d6df78 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
4dd3826a67dfff66ce9c90027d61a9012c0a15e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
62d7e5534486b53634de332875c5fd5d336c29b4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 2a39d2d32807a51346baad28b04d87670381b6d5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
41855e171eaa5bf8da638bc62bce3d0d49dc4bae 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
c63ee5f79b4f9fc356f033960e0af1a7b0058038 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 1378a01f44ef774a15f769460833064c6305b2d6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 2a0c5ca92f3c7431f3c399f309a538f47eb27597 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 85a42f945624c3ca468790772f52363b4064d8fc 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 d414b1405b7672767196b3eaad02baa516169288 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
8400fe98411ed07bd525a51a223fc35423136efb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 30dc1b9da2002689b8b1917f46ae3ca24194f3be 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java 
af04a51b5536550b2d2f7d3e008cf2b2dea607d4 
  ql/src/java/org/apache/hadoop/hive/llap/LlapHiveUtils.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
241a3001e6e0002377736d6d0e820fde004b0bac 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 
210c987b7f580dacda5bdb487af9cf234a738b79 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a9a9f101948a970e0dbf2f77eeb6f688a88d1cbd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 61e2556b08fe4247f35673f24378505ada20a605 
  storage-api/src/java/org/apache/hadoop/hive/common/io/CacheTag.java 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 
2ac0a18a5026e76e65c3c3a8b81d5a844c472ed2 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
d7de3619380d24a1aeea2bac9a66485d7d468517 


Diff: https://reviews.apache.org/r/71575/diff/1/


Testing
---


Thanks,

Adam Szita

[jira] [Created] (HIVE-22286) Disable flaky TestBuddyAllocator#testMTT test

2019-10-02 Thread John Sherman (Jira) 
John Sherman created HIVE-22286:
---

 Summary: Disable flaky TestBuddyAllocator#testMTT test
 Key: HIVE-22286
 URL: https://issues.apache.org/jira/browse/HIVE-22286
 Project: Hive
  Issue Type: Task
Affects Versions: 4.0.0
Reporter: John Sherman
Assignee: John Sherman
 Fix For: 4.0.0


TestBuddyAllocator#TestMTT has been flaky for awhile, failing 20-30% of the 
time. Disabling the test until HIVE-22175 is reviewed and committed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22285) CAST function does not immediately return null if conversion does not succeed

2019-10-02 Thread Eric Richardson (Jira) 
Eric Richardson created HIVE-22285:
--

 Summary: CAST function does not immediately return null if 
conversion does not succeed
 Key: HIVE-22285
 URL: https://issues.apache.org/jira/browse/HIVE-22285
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0
 Environment: HDP 3.1.0
Hive 3.1.0.3.1.0.0-78
Query via DAS 1.3.0 or beeline 3.1.0.3.1.0.0-78
Reporter: Eric Richardson


When cast is called on a column (or derived column) and the cast does not 
succeed it is supposed to return null.  Which is what it returns at the end of 
the query, but some evaluations within the query do not evaluate to null, 
unless the predicate of the cast is null.

ex:
{code:sql}
cast('something that is not a date' as date){code}
will return null

 
{code:sql}
cast('something that is not a date' as date) is NOT null{code}
will return true (which is wrong)

 
{code:sql}
cast(null as date) is NOT null
{code}
will return false (which is correct)

 

full example to demonstrate problem
{code:sql}
CREATE TABLE IF NOT EXISTS castTest (rowId BIGINT, dateString STRING, 
expectedResult BOOLEAN) STORED AS ORC;

TRUNCATE TABLE castTest;

INSERT INTO TABLE castTest VALUES(0,'2019-10-02',true);
INSERT INTO TABLE castTest VALUES(1,null,false);
INSERT INTO TABLE castTest VALUES(2,'  ',false);
INSERT INTO TABLE castTest VALUES(3,'',false);
INSERT INTO TABLE castTest VALUES(4,'not a date',false);

SELECT
  rowId,
  dateString,
  CAST(dateString AS DATE) as dateDate,
  CAST(dateString AS DATE) IS NOT NULL AS actualResult,
  expectedResult,
  nullIf(CAST(dateString AS DATE),null) IS NOT NULL AS workAroundResult
FROM castTest;

DROP TABLE IF EXISTS castTest;

{code}
 returns
{code:sql}
ROWID  DATESTRING  DATEDATEACTUALRESULT  EXPECTEDRESULT  WORKAROUNDRESULT
0  2019-10-02  2019-10-02  true  truetrue
1  nullnullfalse false   false
2  nulltrue  false   false
3  ""  nulltrue  false   false
4  not a date  nulltrue  false   false
{code}
Expected result is true if the date string was a valid convertible date string, 
false otherwise.

 Query (incorrectly) returns true for actual result when dateString is empty, 
whitespace, or characters.

Query only returns correct result when dateString is convertible or null.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics

2019-10-02 Thread Jira 
Ádám Szita created HIVE-22284:
-

 Summary: Improve LLAP CacheContentsTracker to collect and display 
correct statistics
 Key: HIVE-22284
 URL: https://issues.apache.org/jira/browse/HIVE-22284
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Ádám Szita
Assignee: Ádám Szita


When keeping track of which buffers correspond to what Hive objects, 
CacheContentsTracker relies on cache tags.

Currently a tag is a simple String that ideally holds DB and table name, and a 
partition spec concatenated by . and / . The information here is derived from 
the Path of the file that is getting cached. Needless to say sometimes this 
produces a wrong tag especially for external tables.

Also there's a bug when calculating aggregated stats for a 'parent' tag 
(corresponding to the table of the partition) because the overall maxCount and 
maxSize do not add up to the sum of those in the partitions. This happens when 
buffers get removed from the cache.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22283) Aggregate Function on partition column scans Full Table

2019-10-02 Thread Vikas Tarvecha (Jira) 
Vikas Tarvecha created HIVE-22283:
-

 Summary: Aggregate Function on partition column scans Full Table
 Key: HIVE-22283
 URL: https://issues.apache.org/jira/browse/HIVE-22283
 Project: Hive
  Issue Type: Wish
Reporter: Vikas Tarvecha


While running hive queries like "SELECT max(date_partition) FROM 
VERY_LARGE_FACT_TBL", where VERY_LARGE_FACT_TBL is partitioned by column 
'date_partition', a full table scan is required. I have observed that if the 
table contains large amount of data, thousands of mappers are initiated. I wish 
hive could just interact with metastore to pull simple things like max, min or 
distinct of partition column.  I am using Hive version 2.3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71561: HIVE-22250

2019-10-02 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71561/
---

(Updated Oct. 2, 2019, 2:52 p.m.)


Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet 
Garg.


Bugs: HIVE-22250
https://issues.apache.org/jira/browse/HIVE-22250


Repository: hive-git


Description (updated)
---

Describe function does not provide description for rank functions
=
The `DESCRIBE FUNCTION` command gets the description of a function from the 
`@Description` annotations `value` field. If an UDF is annotated with the 
`@WindowFunctionDescription` hive prints 
```
There is no documentation for function 
```
Even if the description is present in the `@WindowFunctionDescription` 
annotation.

This patch moves the `@WindowFunctionDescription.Description` field to a 
separate annotation and provide the `@Description` annotation if both 
annotations are missing.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java fc2a0e1970 
  ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionDescription.java 
511d9641c3 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java a7f4bf1fcc 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 8f4ec3b1ef 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 7a0145243d 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 451b45fbbc 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 4fe9c323cc 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java b31eeb08a0 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 315789c1c1 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 
a8bcc972bb 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCumeDist.java 
70541fe565 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFDenseRank.java 
30bfd2bb8c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFFirstValue.java 
b8b7d8e6da 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLag.java 
e0edbb42af 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLastValue.java 
dadec3b793 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLead.java 
e678278b8b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java 
8b2812d5bc 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentRank.java 
1a7c94431b 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileCont.java
 e7e4fda6ea 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileDisc.java
 d7c295cb11 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRank.java 
a28def73a1 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFRowNumber.java 
41a3e582ec 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
c5c73835af 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBucketNumber.java 
472cc85047 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEpochMilli.java 
d8e822ae97 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 
733fe63e80 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSurrogateKey.java 
1372b60724 
  ql/src/test/queries/clientpositive/desc_function.q d055d9ca03 
  ql/src/test/results/clientpositive/desc_function.q.out 1f804bba60 


Diff: https://reviews.apache.org/r/71561/diff/2/

Changes: https://reviews.apache.org/r/71561/diff/1-2/


Testing
---

Added test cases to `desc_function.q`:
```
DESCRIBE FUNCTION dense_rank;
DESCRIBE FUNCTION EXTENDED dense_rank;
```


Thanks,

Krisztian Kasa

Re: Review Request 71574: HIVE-22212: Implement append partition related methods on temporary tables

2019-10-02 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71574/#review218019
---


Ship it!




Ship It!

- Peter Vary


On okt. 2, 2019, 11:49 de, Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71574/
> ---
> 
> (Updated okt. 2, 2019, 11:49 de)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22212: Implement append partition related methods on temporary tables
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  2467ee3cfb9af0d64653ad7a012ee6d3e68d6674 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAppendPartitionTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAppendPartitions.java
>  e2593fecdce67a2874a14065c9247428ae8852d8 
> 
> 
> Diff: https://reviews.apache.org/r/71574/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication

2019-10-02 Thread Denys Kuzmenko (Jira) 
Denys Kuzmenko created HIVE-22282:
-

 Summary: Obtain LLAP delegation token only when LLAP is configured 
for Kerberos authentication
 Key: HIVE-22282
 URL: https://issues.apache.org/jira/browse/HIVE-22282
 Project: Hive
  Issue Type: Bug
Reporter: Denys Kuzmenko






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 71574: HIVE-22212: Implement append partition related methods on temporary tables

2019-10-02 Thread Laszlo Pinter via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71574/
---

Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.


Repository: hive-git


Description
---

HIVE-22212: Implement append partition related methods on temporary tables


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
2467ee3cfb9af0d64653ad7a012ee6d3e68d6674 
  
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAppendPartitionTempTable.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAppendPartitions.java
 e2593fecdce67a2874a14065c9247428ae8852d8 


Diff: https://reviews.apache.org/r/71574/diff/1/


Testing
---


Thanks,

Laszlo Pinter

[jira] [Created] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"

2019-10-02 Thread Krisztian Kasa (Jira) 
Krisztian Kasa created HIVE-22281:
-

 Summary: Create table statement fails with "not supported NULLS 
LAST for ORDER BY in ASC order"
 Key: HIVE-22281
 URL: https://issues.apache.org/jira/browse/HIVE-22281
 Project: Hive
  Issue Type: Bug
  Components: Parser
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa
 Fix For: 4.0.0


{code}
CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING, 
sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT, op_code 
STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY (k1, 
change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
{code}
{code}
Error while compiling statement: FAILED: SemanticException create/alter table: 
not supported NULLS LAST for ORDER BY in ASC order
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71550: HIVE-22137: Implement alter/rename partition related methods on temporary tables

2019-10-02 Thread Peter Vary via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71550/#review218017
---


Ship it!




Just one minor comment, we can fix that in a follow-up jira


ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
Lines 183-224 (patched)


I think in a follow-up jira we should move these test to the original 
TestAlterPartition test.


- Peter Vary


On szept. 26, 2019, 4:54 du, Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71550/
> ---
> 
> (Updated szept. 26, 2019, 4:54 du)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22137: Implement alter/rename partition related methods on temporary 
> tables
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
> c84c3ef595a5f26232d4f003e46f74bb14a7ec99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a5b16d12a514439046929dde031bbe8e80f71a28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
> fa6dddcbadec59579357b9bc3d4ea42e44a1ca6f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  6c7d80ea576d6b10af191d8ceb158c20b1f70b46 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  4fc3688f2e5c06e254e2177dbf142b424dcfddd8 
> 
> 
> Diff: https://reviews.apache.org/r/71550/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-22280) Q tests for partitioned temporary tables

2019-10-02 Thread Laszlo Pinter (Jira) 
Laszlo Pinter created HIVE-22280:


 Summary: Q tests for partitioned temporary tables
 Key: HIVE-22280
 URL: https://issues.apache.org/jira/browse/HIVE-22280
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22279) Enable temporary table partitioning

2019-10-02 Thread Laszlo Pinter (Jira) 
Laszlo Pinter created HIVE-22279:


 Summary: Enable temporary table partitioning
 Key: HIVE-22279
 URL: https://issues.apache.org/jira/browse/HIVE-22279
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


In order to have full support for partitioned temporary tables, some checks 
from the SemanticAnalyzer must be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71550: HIVE-22137: Implement alter/rename partition related methods on temporary tables

2019-10-02 Thread Laszlo Pinter via Review Board 


> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java
> > Lines 208-210 (patched)
> > 
> >
> > What happens when one of the alter partition fails only? How does it 
> > handled in the HMS handled tables?

I changed this part of the code. Now if this fails, the whole operation is 
reverted, ensuring transactional like behaviour.


> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
> > Lines 195 (patched)
> > 
> >
> > What was the original error on HMS version?
> > Why it is different here?

Changed the exception to InvalidOperationException, and added comment to 
IMetaStoreClient documentation.


> On Sept. 26, 2019, 5:24 p.m., Peter Vary wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
> > Lines 201 (patched)
> > 
> >
> > What was the original error on HMS version?
> > Why it is different here?

The original error is NullPointerException or TProtocolException, depending on 
what HMS implementation is in place.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71550/#review217953
---


On Sept. 26, 2019, 4:54 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71550/
> ---
> 
> (Updated Sept. 26, 2019, 4:54 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22137: Implement alter/rename partition related methods on temporary 
> tables
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionTree.java 
> c84c3ef595a5f26232d4f003e46f74bb14a7ec99 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  a5b16d12a514439046929dde031bbe8e80f71a28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/TempTable.java 
> fa6dddcbadec59579357b9bc3d4ea42e44a1ca6f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestSessionHiveMetastoreClientAlterPartitionsTempTable.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  6c7d80ea576d6b10af191d8ceb158c20b1f70b46 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestAlterPartitions.java
>  4fc3688f2e5c06e254e2177dbf142b424dcfddd8 
> 
> 
> Diff: https://reviews.apache.org/r/71550/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 71571: HIVE-22240

2019-10-02 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71571/
---

(Updated Oct. 2, 2019, 7:41 a.m.)


Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet 
Garg.


Bugs: HIVE-22240
https://issues.apache.org/jira/browse/HIVE-22240


Repository: hive-git


Description
---

Function percentile_cont fails when array parameter passed
==
The method `GenericUDAFPercentileCont.iterate` expect a `HiveDecimalWritable` 
parameter. However array parameters are converted to  
`ArrayList` before they passed. This leads to a 
`ClassCastException`.


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileCont.java
 e7e4fda6ea 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileDisc.java
 d7c295cb11 
  ql/src/test/queries/clientpositive/udaf_percentile_cont.q 7a1ac5c968 
  ql/src/test/queries/clientpositive/udaf_percentile_disc.q 6d93b3418a 
  ql/src/test/results/clientpositive/udaf_percentile_cont.q.out a2dae4a06e 
  ql/src/test/results/clientpositive/udaf_percentile_disc.q.out 3ef0cf9874 


Diff: https://reviews.apache.org/r/71571/diff/2/

Changes: https://reviews.apache.org/r/71571/diff/1-2/


Testing
---

Existing q tests are extended with calling percentile_cont/disc with array 
parameters


Thanks,

Krisztian Kasa