date:20191010

[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949156#comment-16949156
 ] 

Hive QA commented on HIVE-21407:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982695/HIVE-21407.8.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18946/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18946/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18946/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12982695/HIVE-21407.8.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982695 - PreCommit-HIVE-Build

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, 
> HIVE-21407.7.patch, HIVE-21407.8.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread Sankar Hariappan (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949153#comment-16949153
 ] 

Sankar Hariappan commented on HIVE-22319:
-

+1

> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22319.01.patch
>
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22308) Add missing support of Azure Blobstore schemes

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949138#comment-16949138
 ] 

Hive QA commented on HIVE-22308:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982686/HIVE-22308.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17518 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18945/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18945/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18945/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982686 - PreCommit-HIVE-Build

> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22308.02.patch, HIVE-22308.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't 
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22308) Add missing support of Azure Blobstore schemes

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949116#comment-16949116
 ] 

Hive QA commented on HIVE-22308:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 31s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18945/dev-support/hive-personality.sh
 |
| git revision | master / db661ff |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18945/yetus/patch-asflicense-problems.txt
 |
| modules | C: common U: common |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18945/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22308.02.patch, HIVE-22308.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't 
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949102#comment-16949102
 ] 

Hive QA commented on HIVE-22298:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982688/HIVE-22298.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 17518 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=184)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate (batchId=335)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate2 (batchId=335)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedOrc (batchId=335)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.dynamicallyConvertExternalToManagedTable
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testBootstrapLoadMigrationManagedToAcid
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testBootstrapLoadMigrationToAcidWithMoveOptimization
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testIncrementalLoadMigrationManagedToAcid
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testIncrementalLoadMigrationManagedToAcidAllOp
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testIncrementalLoadMigrationManagedToAcidFailure
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testIncrementalLoadMigrationManagedToAcidFailurePart
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.testIncrementalLoadMigrationToAcidWithMoveOptimization
 (batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
 (batchId=272)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.dynamicallyConvertExternalToManagedTable
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testBootstrapLoadMigrationManagedToAcid
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testBootstrapLoadMigrationToAcidWithMoveOptimization
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcid
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidAllOp
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailure
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailurePart
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationToAcidWithMoveOptimization
 (batchId=256)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.testConcurrentOpDuringBootStrapDumpCreateTableReplay
 (batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.testConcurrentOpDuringBootStrapDumpInsertOverwrite
 (batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.testConcurrentOpDuringBootStrapDumpInsertReplay
 (batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.testIncLoadPenFlagPropAlterDB
 (batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.testIncLoadPenFlagWithMoveOptimization
 (batchId=266)
org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=310)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18944/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18944/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18944/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982688 - PreCommit-HIVE-Build

> Allow Llap IO cache for reading tables without delete delta
> ---
>
> Key: HIVE-22298
> URL: https://issues.apache.org/jira/browse/HIVE-22298
> Project: Hive
>  Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22298.2.patch, HIVE-22298.patch
>
>
> Since HIVE-19985 we

[jira] [Commented] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949070#comment-16949070
 ] 

Hive QA commented on HIVE-22298:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
9s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
48s{color} | {color:blue} llap-server in master has 90 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 55 unchanged - 2 
fixed = 55 total (was 57) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} The patch llap-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18944/dev-support/hive-personality.sh
 |
| git revision | master / db661ff |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18944/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql llap-server U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18944/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow Llap IO cache for reading tables without delete delta
> ---
>
> Key: HIVE-22298
> URL: https://issues.apache.org/jira/browse/HIVE-22298
> Project: Hive
>  Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22298.2.patch, HIVE-22298.patch
>
>
> Since HIVE-19985 we can use Llap IO cache for queries which are not 
> explicitly reading ROW_ID even for original files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949040#comment-16949040
 ] 

Hive QA commented on HIVE-22217:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982531/HIVE-22217.01.branch-3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 149 failed/errored test(s), 14438 tests 
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed 
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out) 
(batchId=232)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=276)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed 
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestChainFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=232)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestCloseableThreadLocal - did not produce a TEST-*.xml file (likely timed out) 
(batchId=335)
TestCustomQueryFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=232)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed 
out) (batchId=239)
TestDatabases - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=276)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed 
out) (batchId=240)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestGetPartitionsUsingProjectionAndFilterSpecs - did not produce a TEST-*.xml 
file (likely timed out) (batchId=230)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestGroupFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=232)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out) 
(batchId=239)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out) 
(batchId=239)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=237)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed 
out) (batchId=239)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely 
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely 
timed out) (batchId=237)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out) 
(batchId=239)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out) 
(batchId=239)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file 
(likely timed out) (batchId=234)
TestHiveMetaToolCommandLine - did not produce a TEST-*.xml file (likely timed 
out) (batchId=232)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out) 
(batchId=228)
TestHmsServerAuthorization - did not produce a TEST-*.xml file (likely timed 
out) (batchId=237)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out) 
(batchId=240)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out) 
(batchId=239)
TestHyperLogLogMerge - did not produce a

[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949011#comment-16949011
 ] 

Hive QA commented on HIVE-22217:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 12s{color} 
| {color:red} 
/data/hiveptest/logs/PreCommit-HIVE-Build-18943/patches/PreCommit-HIVE-Build-18943.patch
 does not apply to master. Rebase required? Wrong Branch? See 
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. 
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18943/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Better Logging for Hive JAR Reload
> --
>
> Key: HIVE-22217
> URL: https://issues.apache.org/jira/browse/HIVE-22217
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.2.0, 2.3.6
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22217.01.branch-3.patch, HIVE-22217.1.patch, 
> HIVE-22217.branch3.1.patch
>
>
> Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult.
> Add logging to at least confirm which JAR files are being loaded.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar

2019-10-10 Thread Ramesh Kumar Thangarajan (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramesh Kumar Thangarajan updated HIVE-22315:

Attachment: HIVE-22315.2.patch
Status: Patch Available  (was: Open)

> Support Decimal64 column division with decimal64 scalar
> ---
>
> Key: HIVE-22315
> URL: https://issues.apache.org/jira/browse/HIVE-22315
> Project: Hive
>  Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch
>
>
> Currently division operation is not supported for Decimal64 column. This Jira 
> will take care of supporting decimal64 column division with a decimal64 
> scalar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar

2019-10-10 Thread Ramesh Kumar Thangarajan (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramesh Kumar Thangarajan updated HIVE-22315:

Status: Open  (was: Patch Available)

> Support Decimal64 column division with decimal64 scalar
> ---
>
> Key: HIVE-22315
> URL: https://issues.apache.org/jira/browse/HIVE-22315
> Project: Hive
>  Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22315.1.patch
>
>
> Currently division operation is not supported for Decimal64 column. This Jira 
> will take care of supporting decimal64 column division with a decimal64 
> scalar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

2019-10-10 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21344:
---
Attachment: HIVE-21344.05.patch

> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21344.01.patch, HIVE-21344.02.patch, 
> HIVE-21344.03.patch, HIVE-21344.04.patch, HIVE-21344.05.patch, 
> HIVE-21344.patch, calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to 
> metastore to verify that the materializations exist, whether they are 
> outdated or not, etc. Since this is only useful for active-active HS2 
> deployments, we could take a less aggressive approach and check this 
> information only after rewriting has been triggered. In addition, we could 
> refresh the information in the HS2 registry periodically in a background 
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
> // to force the materialization contents to be up-to-date, as this is not a 
> rebuild, and
> // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan), 
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21344?focusedWorklogId=326625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326625
 ]

ASF GitHub Bot logged work on HIVE-21344:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 22:54
Start Date: 10/Oct/19 22:54
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on issue #749: HIVE-21344
URL: https://github.com/apache/hive/pull/749#issuecomment-540831437
 
 
   @vineetgarg02 , I addressed your comments. Can you take another look? Thanks
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326625)
Time Spent: 50m  (was: 40m)

> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21344.01.patch, HIVE-21344.02.patch, 
> HIVE-21344.03.patch, HIVE-21344.04.patch, HIVE-21344.05.patch, 
> HIVE-21344.patch, calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to 
> metastore to verify that the materializations exist, whether they are 
> outdated or not, etc. Since this is only useful for active-active HS2 
> deployments, we could take a less aggressive approach and check this 
> information only after rewriting has been triggered. In addition, we could 
> refresh the information in the HS2 registry periodically in a background 
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
> // to force the materialization contents to be up-to-date, as this is not a 
> rebuild, and
> // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan), 
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21344?focusedWorklogId=326622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326622
 ]

ASF GitHub Bot logged work on HIVE-21344:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 22:49
Start Date: 10/Oct/19 22:49
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #749: HIVE-21344
URL: https://github.com/apache/hive/pull/749#discussion_r333767332
 
 

 ##
 File path: 
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 ##
 @@ -159,15 +165,36 @@ public void run() {
 SessionState ss = new SessionState(db.getConf());
 ss.setIsHiveServerQuery(true); // All is served from HS2, we do not 
need e.g. Tez sessions
 SessionState.start(ss);
-final boolean cache = !db.getConf()
-
.get(HiveConf.ConfVars.HIVE_SERVER2_MATERIALIZED_VIEWS_REGISTRY_IMPL.varname).equals("DUMMY");
-for (Table mv : db.getAllMaterializedViewObjectsForRewriting()) {
-  addMaterializedView(db.getConf(), mv, OpType.LOAD, cache);
+if (initialized.get()) {
+  for (Table mvTable : db.getAllMaterializedViewObjectsForRewriting()) 
{
 
 Review comment:
   Good idea! 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326622)
Time Spent: 40m  (was: 0.5h)

> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21344.01.patch, HIVE-21344.02.patch, 
> HIVE-21344.03.patch, HIVE-21344.04.patch, HIVE-21344.patch, 
> calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to 
> metastore to verify that the materializations exist, whether they are 
> outdated or not, etc. Since this is only useful for active-active HS2 
> deployments, we could take a less aggressive approach and check this 
> information only after rewriting has been triggered. In addition, we could 
> refresh the information in the HS2 registry periodically in a background 
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
> // to force the materialization contents to be up-to-date, as this is not a 
> rebuild, and
> // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan), 
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21344?focusedWorklogId=326610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326610
 ]

ASF GitHub Bot logged work on HIVE-21344:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 22:32
Start Date: 10/Oct/19 22:32
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #749: HIVE-21344
URL: https://github.com/apache/hive/pull/749#discussion_r333763093
 
 

 ##
 File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
 ##
 @@ -1612,6 +1612,150 @@ public Table 
apply(org.apache.hadoop.hive.metastore.api.Table table) {
 }
   }
 
+  /**
+   * Get the materialized views that have been enabled for rewriting from the
+   * cache (registry). It will preprocess them to discard those that are
+   * outdated and augment those that need to be augmented, e.g., if incremental
+   * rewriting is enabled.
+   *
+   * @return the list of materialized views available for rewriting from the 
registry
+   * @throws HiveException
+   */
+  public List 
getPreprocessedMaterializedViewsFromRegistry(
+  List tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+// From cache
+List materializedViews =
+HiveMaterializedViewsRegistry.get().getRewritingMaterializedViews();
+if (materializedViews.isEmpty()) {
+  // Bail out: empty list
+  return new ArrayList<>();
+}
+// Add to final result
+return filterAugmentMaterializedViews(materializedViews, tablesUsed, 
txnMgr);
+  }
+
+  private List 
filterAugmentMaterializedViews(List materializedViews,
+List tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+final String validTxnsList = conf.get(ValidTxnList.VALID_TXNS_KEY);
+final ValidTxnWriteIdList currentTxnWriteIds = 
txnMgr.getValidWriteIds(tablesUsed, validTxnsList);
+final boolean tryIncrementalRewriting =
+HiveConf.getBoolVar(conf, 
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_INCREMENTAL);
+final long defaultTimeWindow =
+HiveConf.getTimeVar(conf, 
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW,
+TimeUnit.MILLISECONDS);
+try {
+  // Final result
+  List result = new ArrayList<>();
+  for (RelOptMaterialization materialization : materializedViews) {
+final RelNode viewScan = materialization.tableRel;
+final Table materializedViewTable;
+if (viewScan instanceof Project) {
+  // There is a Project on top (due to nullability)
+  materializedViewTable = ((RelOptHiveTable) 
viewScan.getInput(0).getTable()).getHiveTableMD();
+} else {
+  materializedViewTable = ((RelOptHiveTable) 
viewScan.getTable()).getHiveTableMD();
+}
+final Boolean outdated = 
isOutdatedMaterializedView(materializedViewTable, currentTxnWriteIds,
+defaultTimeWindow, tablesUsed, false);
+if (outdated == null) {
+  continue;
+}
+
+final CreationMetadata creationMetadata = 
materializedViewTable.getCreationMetadata();
+if (outdated) {
+  // The MV is outdated, see whether we should consider it for 
rewriting or not
+  if (!tryIncrementalRewriting) {
+LOG.debug("Materialized view " + 
materializedViewTable.getFullyQualifiedName() +
+" ignored for rewriting as its contents are outdated");
+continue;
+  }
+  // We will rewrite it to include the filters on transaction list
+  // so we can produce partial rewritings.
+  // This would be costly since we are doing it for every materialized 
view
+  // that is outdated, but it only happens for more than one 
materialized view
+  // if rewriting with outdated materialized views is enabled 
(currently
+  // disabled by default).
+  materialization = augmentMaterializationWithTimeInformation(
+  materialization, validTxnsList, new ValidTxnWriteIdList(
+  creationMetadata.getValidTxnList()));
+}
+result.add(materialization);
+  }
+  return result;
+} catch (Exception e) {
+  throw new HiveException(e);
+}
+  }
+
+  /**
+   * Validate that the materialized views retrieved from registry are still 
up-to-date.
+   * For those that are not, the method loads them from the metastore into the 
registry.
+   *
+   * @return true if they are up-to-date, otherwise false
+   * @throws HiveException
+   */
+  public boolean validateMaterializedViewsFromRegistry(List 
cachedMaterializedViewTables,
+  List tablesUsed, HiveTxnManager txnMgr) throws HiveException {
+final long defaultTimeWindow =
+HiveConf.getTimeVar(conf, 
HiveConf.ConfVars.HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW,
+TimeUnit.MILLISECONDS);
+final String validTxnsList =

[jira] [Updated] (HIVE-22118) Log the table name while skipping the compaction because it's sorted table/partitions

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-22118:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Log the table name while skipping the compaction because it's sorted 
> table/partitions
> -
>
> Key: HIVE-22118
> URL: https://issues.apache.org/jira/browse/HIVE-22118
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22118.patch
>
>
> for debugging perspective it's good if we log the full table name while 
> skipping the table for compaction otherwise it's tedious to know why the 
> compaction is not happening for the target table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22118) Log the table name while skipping the compaction because it's sorted table/partitions

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-22118:
--
Fix Version/s: 4.0.0
   Status: In Progress  (was: Patch Available)

> Log the table name while skipping the compaction because it's sorted 
> table/partitions
> -
>
> Key: HIVE-22118
> URL: https://issues.apache.org/jira/browse/HIVE-22118
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22118.patch
>
>
> for debugging perspective it's good if we log the full table name while 
> skipping the table for compaction otherwise it's tedious to know why the 
> compaction is not happening for the target table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22118) Log the table name while skipping the compaction because it's sorted table/partitions

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-22118:
--
Status: Patch Available  (was: In Progress)

> Log the table name while skipping the compaction because it's sorted 
> table/partitions
> -
>
> Key: HIVE-22118
> URL: https://issues.apache.org/jira/browse/HIVE-22118
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22118.patch
>
>
> for debugging perspective it's good if we log the full table name while 
> skipping the table for compaction otherwise it's tedious to know why the 
> compaction is not happening for the target table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948906#comment-16948906
 ] 

Hive QA commented on HIVE-22319:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982677/HIVE-22319.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17518 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18942/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18942/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18942/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982677 - PreCommit-HIVE-Build

> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22319.01.patch
>
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948903#comment-16948903
 ] 

Hive QA commented on HIVE-22319:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
15s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 3 new + 25 unchanged - 0 fixed 
= 28 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  
0s{color} | {color:red} root: The patch generated 3 new + 25 unchanged - 0 
fixed = 28 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18942/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18942/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18942/yetus/diff-checkstyle-root.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18942/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18942/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22319.01.patch
>
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22325) variable expansion doesn't work in beeline-site.xml

2019-10-10 Thread Allan Espinosa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Espinosa updated HIVE-22325:
--
Affects Version/s: 3.1.0
   3.1.1

> variable expansion doesn't work in beeline-site.xml
> ---
>
> Key: HIVE-22325
> URL: https://issues.apache.org/jira/browse/HIVE-22325
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 3.1.0, 3.1.1, 3.1.2
>Reporter: Allan Espinosa
>Assignee: Allan Espinosa
>Priority: Major
>
> I have a default jdbc connection string and I want to build on top on it to 
> have customized connections like setting custom queue names.  
> {code}
> $ cat .beeline/beeline-site.xml
>   http://www.w3.org/2001/XInclude;>
> 
>   beeline.hs2.jdbc.url.base
>   jdbc:hive2://localhost/
> 
> 
>   beeline.hs2.jdbc.url.myqueue
>   ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue
> 
>   
> $ beeline -c myqueue
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue from beeline-site.xml
> Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
> beeline>
> {code}
> Relevant code is found in 
> https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java#L94
> Entry#getValue() skips the variable expansion .  Using 
> Configuration#get(key) would make this work.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22326) StreamingV2: Fail streaming ingests if columns with default constraints are not provided

2019-10-10 Thread Gopal Vijayaraghavan (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal Vijayaraghavan updated HIVE-22326:

Component/s: Streaming

> StreamingV2: Fail streaming ingests if columns with default constraints are 
> not provided
> 
>
> Key: HIVE-22326
> URL: https://issues.apache.org/jira/browse/HIVE-22326
> Project: Hive
>  Issue Type: Bug
>  Components: Streaming
>Reporter: Gopal Vijayaraghavan
>Priority: Major
>
> If a column has a default constraint, the StreamingV2 does not run the 
> corresponding UDF (& in some cases cannot run one, like SURROGATE_KEY).
> Fail visibly for that scenario by scenario, rather than allowing DEFAULT to 
> be ignored.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22325) variable expansion doesn't work in beeline-site.xml

2019-10-10 Thread Rajkumar Singh (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948852#comment-16948852
 ] 

Rajkumar Singh commented on HIVE-22325:
---

sound duplicate of https://issues.apache.org/jira/browse/HIVE-22317

> variable expansion doesn't work in beeline-site.xml
> ---
>
> Key: HIVE-22325
> URL: https://issues.apache.org/jira/browse/HIVE-22325
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 3.1.2
>Reporter: Allan Espinosa
>Assignee: Allan Espinosa
>Priority: Major
>
> I have a default jdbc connection string and I want to build on top on it to 
> have customized connections like setting custom queue names.  
> {code}
> $ cat .beeline/beeline-site.xml
>   http://www.w3.org/2001/XInclude;>
> 
>   beeline.hs2.jdbc.url.base
>   jdbc:hive2://localhost/
> 
> 
>   beeline.hs2.jdbc.url.myqueue
>   ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue
> 
>   
> $ beeline -c myqueue
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue from beeline-site.xml
> Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
> beeline>
> {code}
> Relevant code is found in 
> https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java#L94
> Entry#getValue() skips the variable expansion .  Using 
> Configuration#get(key) would make this work.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22325) variable expansion doesn't work in beeline-site.xml

2019-10-10 Thread Allan Espinosa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22325 started by Allan Espinosa.
-
> variable expansion doesn't work in beeline-site.xml
> ---
>
> Key: HIVE-22325
> URL: https://issues.apache.org/jira/browse/HIVE-22325
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 3.1.2
>Reporter: Allan Espinosa
>Assignee: Allan Espinosa
>Priority: Major
>
> I have a default jdbc connection string and I want to build on top on it to 
> have customized connections like setting custom queue names.  
> {code}
> $ cat .beeline/beeline-site.xml
>   http://www.w3.org/2001/XInclude;>
> 
>   beeline.hs2.jdbc.url.base
>   jdbc:hive2://localhost/
> 
> 
>   beeline.hs2.jdbc.url.myqueue
>   ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue
> 
>   
> $ beeline -c myqueue
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue from beeline-site.xml
> Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
> beeline>
> {code}
> Relevant code is found in 
> https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java#L94
> Entry#getValue() skips the variable expansion .  Using 
> Configuration#get(key) would make this work.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22325) variable expansion doesn't work in beeline-site.xml

2019-10-10 Thread Allan Espinosa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Espinosa reassigned HIVE-22325:
-


> variable expansion doesn't work in beeline-site.xml
> ---
>
> Key: HIVE-22325
> URL: https://issues.apache.org/jira/browse/HIVE-22325
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 3.1.2
>Reporter: Allan Espinosa
>Assignee: Allan Espinosa
>Priority: Major
>
> I have a default jdbc connection string and I want to build on top on it to 
> have customized connections like setting custom queue names.  
> {code}
> $ cat .beeline/beeline-site.xml
>   http://www.w3.org/2001/XInclude;>
> 
>   beeline.hs2.jdbc.url.base
>   jdbc:hive2://localhost/
> 
> 
>   beeline.hs2.jdbc.url.myqueue
>   ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue
> 
>   
> $ beeline -c myqueue
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.base}?tez.queue.name=myqueue from beeline-site.xml
> Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
> beeline>
> {code}
> Relevant code is found in 
> https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java#L94
> Entry#getValue() skips the variable expansion .  Using 
> Configuration#get(key) would make this work.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22274) Upgrade Calcite version to 1.21.0

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-22274:
--
Labels: pull-request-available  (was: )

> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
>  Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch, 
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch, HIVE-22274.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=326488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326488
 ]

ASF GitHub Bot logged work on HIVE-22274:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 18:18
Start Date: 10/Oct/19 18:18
Worklog Time Spent: 10m 
  Work Description: scarlin-cloudera commented on pull request #809: 
HIVE-22274: upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326488)
Remaining Estimate: 0h
Time Spent: 10m

> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
>  Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch, 
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch, HIVE-22274.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22324) Checkin test output changes due to Calcite 1.21 upgrade

2019-10-10 Thread Steve Carlin (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Carlin reassigned HIVE-22324:
---

Assignee: Steve Carlin

> Checkin test output changes due to Calcite 1.21 upgrade
> ---
>
> Key: HIVE-22324
> URL: https://issues.apache.org/jira/browse/HIVE-22324
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
>
> On the upgrade to Calcite 1.21, CALC-2991 caused a change in some of the 
> planner output.  This initial hive checkin for the upgrade did an override of 
> the RelMdMaxRowCount class to simulate 1.19 behavior.
> This task is to remove the HiveRelMdMaxRowCount class, use the new 1.21 code, 
> and change the q.out files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22323) Fix Desc Table bugs

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948839#comment-16948839
 ] 

Hive QA commented on HIVE-22323:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982678/HIVE-22323.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 329 failed/errored test(s), 17520 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[colstats_all_nulls] 
(batchId=299)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments] 
(batchId=299)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] 
(batchId=299)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart] 
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStats] 
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format] 
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
 (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23]
 (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_clusterby_sortby]
 (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status]
 (batchId=100)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_add_partition]
 (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_column_stats]
 (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_not_sorted] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status]
 (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
 (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
 (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_as_select] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_col_type] 
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_part] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_10] 
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_11] 
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1] 
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2] 
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3] 
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_native] 
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
 (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[basicstat_partval] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bitvector] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[colstats_all_nulls] 
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
 (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_names_with_leading_and_trailing_spaces]
 (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_pruner_multiple_children]
 (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_infinity] 
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp] 
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_tbllvl] 
(batchId=9)

[jira] [Commented] (HIVE-22323) Fix Desc Table bugs

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948805#comment-16948805
 ] 

Hive QA commented on HIVE-22323:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
29s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
51s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
46s{color} | {color:red} ql: The patch generated 1 new + 307 unchanged - 0 
fixed = 308 total (was 307) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
24s{color} | {color:red} ql generated 1 new + 1550 unchanged - 0 fixed = 1551 
total (was 1550) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
0s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total 
(was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 31s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.ddl.table.info.DescTableDesc.COLUMN_STATISTICS_HEADERS
 is a mutable array  At DescTableDesc.java: At DescTableDesc.java:[line 38] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18941/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18941/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18941/yetus/new-findbugs-ql.html
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18941/yetus/diff-javadoc-javadoc-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18941/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18941/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Fix Desc Table bugs
> ---
>
> Key: HIVE-22323
> URL: https://issues.apache.org/jira/browse/HIVE-22323
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22323.01.patch
>
>
> DESC TABLE operation is having the following bugs:
>  # Whole table descs have two headers.
>  # Table column desc has incorrect long header, while the table is transposed 
> having the headers in the first column.
>  # Json formatted data

[jira] [Updated] (HIVE-21954) QTest: support for running qtests on various metastore DBs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-21954:

Issue Type: Improvement  (was: Bug)

> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local 
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch 
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest 
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q 
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be 
> edited manually. My proposal is to introduce a property 
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly. 
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password, 
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro 
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>  
> In the first implementation, I would not start metastore db automatically 
> (which is done be 'mvn verify ...'), but it's still under planning. 
> In the long term, we should consider running this kind of tests in precommit 
> phase, so maybe -Dmetastore.db=postgres could start metastore db 
> automatically. Also we should consider running some qtests on various 
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones 
> instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21954) QTest: support for running qtests on various metastore DBs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-21954:

Component/s: Testing Infrastructure
 Standalone Metastore

> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local 
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch 
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest 
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q 
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be 
> edited manually. My proposal is to introduce a property 
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly. 
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password, 
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro 
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>  
> In the first implementation, I would not start metastore db automatically 
> (which is done be 'mvn verify ...'), but it's still under planning. 
> In the long term, we should consider running this kind of tests in precommit 
> phase, so maybe -Dmetastore.db=postgres could start metastore db 
> automatically. Also we should consider running some qtests on various 
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones 
> instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21954) QTest: support for running qtests on various metastore DBs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-21954:

Fix Version/s: 4.0.0

> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local 
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch 
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest 
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q 
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be 
> edited manually. My proposal is to introduce a property 
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly. 
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password, 
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro 
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>  
> In the first implementation, I would not start metastore db automatically 
> (which is done be 'mvn verify ...'), but it's still under planning. 
> In the long term, we should consider running this kind of tests in precommit 
> phase, so maybe -Dmetastore.db=postgres could start metastore db 
> automatically. Also we should consider running some qtests on various 
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones 
> instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948763#comment-16948763
 ] 

Hive QA commented on HIVE-21407:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982695/HIVE-21407.8.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17534 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18940/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18940/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18940/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982695 - PreCommit-HIVE-Build

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, 
> HIVE-21407.7.patch, HIVE-21407.8.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-21954) QTest: support for running qtests on various metastore DBs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21954 started by László Bodor.
---
> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local 
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch 
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest 
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q 
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be 
> edited manually. My proposal is to introduce a property 
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly. 
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password, 
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro 
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>  
> In the first implementation, I would not start metastore db automatically 
> (which is done be 'mvn verify ...'), but it's still under planning. 
> In the long term, we should consider running this kind of tests in precommit 
> phase, so maybe -Dmetastore.db=postgres could start metastore db 
> automatically. Also we should consider running some qtests on various 
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones 
> instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22317) Beeline-site parser does not handle the variable substitution correctly

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-22317:
--
Summary: Beeline-site parser does not handle the variable substitution 
correctly  (was: Beeline site parser does not handle the variable substitution 
correctly)

> Beeline-site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>  
>  
>  beeline.hs2.jdbc.url.container
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>  
>  
>  
>  beeline.hs2.jdbc.url.default
>  test
>  
>  
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>  
>  
>  beeline.hs2.jdbc.url.llap
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>  
>  
>  {code}
> beeline fail to connect because it does not parse the substituted value 
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision

2019-10-10 Thread Mustafa Iman (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mustafa Iman updated HIVE-14302:

Attachment: HIVE-14302.6.patch
Status: Patch Available  (was: In Progress)

> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14302.2.patch, HIVE-14302.3.patch, 
> HIVE-14302.4.patch, HIVE-14302.5.patch, HIVE-14302.6.patch, HIVE-14302.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the 
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because 
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK  
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
>   TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
>   predicate: (a is not null and true) (type: boolean)
>   Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
>   key expressions: _col0 (type: decimal(11,2))
>   sort order: +
>   Map-reduce partition columns: _col0 (type: decimal(11,2))
>   Join Operator (JOIN_8)
> condition map:
>  Inner Join 0 to 1
> keys:
>   0 _col0 (type: decimal(11,2))
>   1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
>   aggregations: count(1)
>   mode: hash
>   outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the 
> join to be able to compare HiveDecimal as-is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision

2019-10-10 Thread Mustafa Iman (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mustafa Iman updated HIVE-14302:

Status: In Progress  (was: Patch Available)

> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14302.2.patch, HIVE-14302.3.patch, 
> HIVE-14302.4.patch, HIVE-14302.5.patch, HIVE-14302.6.patch, HIVE-14302.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the 
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because 
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK  
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
>   TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
>   predicate: (a is not null and true) (type: boolean)
>   Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
>   key expressions: _col0 (type: decimal(11,2))
>   sort order: +
>   Map-reduce partition columns: _col0 (type: decimal(11,2))
>   Join Operator (JOIN_8)
> condition map:
>  Inner Join 0 to 1
> keys:
>   0 _col0 (type: decimal(11,2))
>   1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
>   aggregations: count(1)
>   mode: hash
>   outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the 
> join to be able to compare HiveDecimal as-is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21114) Create read-only transactions

2019-10-10 Thread Denys Kuzmenko (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-21114:
--
Status: Patch Available  (was: In Progress)

> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21114.1.patch
>
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a 
> read and mark the txn accordingly.  
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any 
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt, 
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this 
> requires figuring out how to determine if a query is a SELECT.  By the time 
> {{Driver.openTransaction();}} is called, we have already parsed the query so 
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to 
> indicate that a txn is read-only and then not allow any statements that can 
> make modifications in this txn.  This should be a different jira.
> cc [~ikryvenko]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21114) Create read-only transactions

2019-10-10 Thread Denys Kuzmenko (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-21114:
--
Attachment: HIVE-21114.1.patch

> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21114.1.patch
>
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a 
> read and mark the txn accordingly.  
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any 
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt, 
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this 
> requires figuring out how to determine if a query is a SELECT.  By the time 
> {{Driver.openTransaction();}} is called, we have already parsed the query so 
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to 
> indicate that a txn is read-only and then not allow any statements that can 
> make modifications in this txn.  This should be a different jira.
> cc [~ikryvenko]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21114) Create read-only transactions

2019-10-10 Thread Denys Kuzmenko (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko updated HIVE-21114:
--
Attachment: (was: HIVE-21114.1.patch)

> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a 
> read and mark the txn accordingly.  
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any 
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt, 
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this 
> requires figuring out how to determine if a query is a SELECT.  By the time 
> {{Driver.openTransaction();}} is called, we have already parsed the query so 
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to 
> indicate that a txn is read-only and then not allow any statements that can 
> make modifications in this txn.  This should be a different jira.
> cc [~ikryvenko]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948718#comment-16948718
 ] 

Hive QA commented on HIVE-21407:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 18 new + 260 unchanged - 12 
fixed = 278 total (was 272) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
17s{color} | {color:red} ql generated 1 new + 1549 unchanged - 1 fixed = 1550 
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 48s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.hive.ql.io.parquet.LeafFilterFactory$BooleanFilterPredicateLeafBuilder.buildPredict(PredicateLeaf$Operator,
 Object, String, TypeInfo)  At LeafFilterFactory.java:then immediately reboxed 
in 
org.apache.hadoop.hive.ql.io.parquet.LeafFilterFactory$BooleanFilterPredicateLeafBuilder.buildPredict(PredicateLeaf$Operator,
 Object, String, TypeInfo)  At LeafFilterFactory.java:[line 139] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18940/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18940/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18940/yetus/new-findbugs-ql.html
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18940/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18940/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, 
> HIVE-21407.7.patch, HIVE-21407.8.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
>

[jira] [Updated] (HIVE-22239) Scale data size using column value ranges

2019-10-10 Thread Jesus Camacho Rodriguez (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-22239:
---
Attachment: HIVE-22239.05.patch

> Scale data size using column value ranges
> -
>
> Key: HIVE-22239
> URL: https://issues.apache.org/jira/browse/HIVE-22239
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, 
> HIVE-22239.03.patch, HIVE-22239.04.patch, HIVE-22239.04.patch, 
> HIVE-22239.05.patch, HIVE-22239.05.patch, HIVE-22239.patch
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Currently, min/max values for columns are only used to determine whether a 
> certain range filter falls out of range and thus filters all rows or none at 
> all. If it does not, we just use a heuristic that the condition will filter 
> 1/3 of the input rows. Instead of using that heuristic, we can use another 
> one that assumes that data will be uniformly distributed across that range, 
> and calculate the selectivity for the condition accordingly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21198) Introduce a database object reference class

2019-10-10 Thread Jesus Camacho Rodriguez (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948708#comment-16948708
 ] 

Jesus Camacho Rodriguez commented on HIVE-21198:


[~dlavati], [~kgyrtkirk], is there any ETA for this issue and HIVE-21151? Thanks

> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.2.patch, 
> HIVE-21198.3.patch, HIVE-21198.4.patch, HIVE-21198.5.patch, 
> HIVE-21198.6.patch, HIVE-21198.7.patch, HIVE-21198.8.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a 
> single string; there are some places where the they travel as 2 separate 
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass 
> these informations together. Making this better is required if we would be 
> wanting to enable dot in tablenames 
> HIVE-16907, HIVE-21151



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22248) Min value for column in stats is not set correctly for some data types

2019-10-10 Thread Jesus Camacho Rodriguez (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948707#comment-16948707
 ] 

Jesus Camacho Rodriguez commented on HIVE-22248:


[~mgergely], would you mind to create a new JIRA for that issue? It makes it 
easier to keep track. Thanks

> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch, 
> HIVE-22248.04.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value 
> stored in the metastore itself, but for some types (e.g. tinyint, smallint, 
> int, bigint, double or float), the min value does not seem to be set 
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948688#comment-16948688
 ] 

Hive QA commented on HIVE-22322:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982673/HIVE-22322.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17518 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18939/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18939/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18939/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982673 - PreCommit-HIVE-Build

> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-22322.1.patch
>
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Marta Kuczora (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-21407:
-
Attachment: HIVE-21407.8.patch

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, 
> HIVE-21407.7.patch, HIVE-21407.8.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948671#comment-16948671
 ] 

Hive QA commented on HIVE-22322:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18939/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18939/yetus/patch-asflicense-problems.txt
 |
| modules | C: llap-server . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18939/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-22322.1.patch
>
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HIVE-22217) Better Logging for Hive JAR Reload

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor reopened HIVE-22217:
-

> Better Logging for Hive JAR Reload
> --
>
> Key: HIVE-22217
> URL: https://issues.apache.org/jira/browse/HIVE-22217
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.2.0, 2.3.6
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22217.01.branch-3.patch, HIVE-22217.1.patch, 
> HIVE-22217.branch3.1.patch
>
>
> Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult.
> Add logging to at least confirm which JAR files are being loaded.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22217) Better Logging for Hive JAR Reload

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22217:

Status: Patch Available  (was: Reopened)

> Better Logging for Hive JAR Reload
> --
>
> Key: HIVE-22217
> URL: https://issues.apache.org/jira/browse/HIVE-22217
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.3.6, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22217.01.branch-3.patch, HIVE-22217.1.patch, 
> HIVE-22217.branch3.1.patch
>
>
> Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult.
> Add logging to at least confirm which JAR files are being loaded.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta

2019-10-10 Thread Peter Vary (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-22298:
--
Attachment: HIVE-22298.2.patch

> Allow Llap IO cache for reading tables without delete delta
> ---
>
> Key: HIVE-22298
> URL: https://issues.apache.org/jira/browse/HIVE-22298
> Project: Hive
>  Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22298.2.patch, HIVE-22298.patch
>
>
> Since HIVE-19985 we can use Llap IO cache for queries which are not 
> explicitly reading ROW_ID even for original files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22308) Add missing support of Azure Blobstore schemes

2019-10-10 Thread David Lavati (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948641#comment-16948641
 ] 

David Lavati commented on HIVE-22308:
-

I've realized the http/s schemes are used only internally and are never 
returned by any FileSystem:getScheme method, so I've removed them.

> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22308.02.patch, HIVE-22308.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't 
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22308) Add missing support of Azure Blobstore schemes

2019-10-10 Thread David Lavati (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Lavati updated HIVE-22308:

Attachment: HIVE-22308.02.patch

> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22308.02.patch, HIVE-22308.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't 
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22308) Add missing support of Azure Blobstore schemes

2019-10-10 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948630#comment-16948630
 ] 

László Bodor commented on HIVE-22308:
-

+1


> Add missing support of Azure Blobstore schemes
> --
>
> Key: HIVE-22308
> URL: https://issues.apache.org/jira/browse/HIVE-22308
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22308.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Azure has been used as a filesystem for Hive, but its various schemes aren't 
> registered under
> {{HiveConf.HIVE_BLOBSTORE_SUPPORTED_SCHEMES.}}
> Found the list of elements in: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemUriSchemes.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948599#comment-16948599
 ] 

Hive QA commented on HIVE-22292:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982667/HIVE-22292.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17518 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.TestFunctionRegistry.testImpliesOrder 
(batchId=345)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18938/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18938/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18938/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982667 - PreCommit-HIVE-Build

> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch
>
>
> {code}
>  ::=
>
>
>   
>  ::=
>   RANK
>   | DENSE_RANK
>   | PERCENT_RANK
>   | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL), 
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22248) Min value for column in stats is not set correctly for some data types

2019-10-10 Thread Miklos Gergely (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948592#comment-16948592
 ] 

Miklos Gergely commented on HIVE-22248:
---

The *ColumnStatsMerger classes are only doing the merge in case of a not 
partitioned table. The partitioned tables are merged by the 
*ColumnStatsAggregator classes, thus they should be fixed as well, by making 
them use the mergers for actual merging of the low / high values.

> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch, 
> HIVE-22248.04.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value 
> stored in the metastore itself, but for some types (e.g. tinyint, smallint, 
> int, bigint, double or float), the min value does not seem to be set 
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HIVE-22248) Min value for column in stats is not set correctly for some data types

2019-10-10 Thread Miklos Gergely (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Gergely reopened HIVE-22248:
---

> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch, 
> HIVE-22248.04.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value 
> stored in the metastore itself, but for some types (e.g. tinyint, smallint, 
> int, bigint, double or float), the min value does not seem to be set 
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22323) Fix Desc Table bugs

2019-10-10 Thread Miklos Gergely (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Gergely updated HIVE-22323:
--
Attachment: HIVE-22323.01.patch

> Fix Desc Table bugs
> ---
>
> Key: HIVE-22323
> URL: https://issues.apache.org/jira/browse/HIVE-22323
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22323.01.patch
>
>
> DESC TABLE operation is having the following bugs:
>  # Whole table descs have two headers.
>  # Table column desc has incorrect long header, while the table is transposed 
> having the headers in the first column.
>  # Json formatted data also has the headers.
>  # Json formatted data doesn't have the column statistics in it.
>  # There is no TestBeeLineDriver test for desc table, thus the actual output 
> is not tested, just some intermediary.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22323) Fix Desc Table bugs

2019-10-10 Thread Miklos Gergely (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Gergely updated HIVE-22323:
--
Status: Patch Available  (was: Open)

> Fix Desc Table bugs
> ---
>
> Key: HIVE-22323
> URL: https://issues.apache.org/jira/browse/HIVE-22323
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22323.01.patch
>
>
> DESC TABLE operation is having the following bugs:
>  # Whole table descs have two headers.
>  # Table column desc has incorrect long header, while the table is transposed 
> having the headers in the first column.
>  # Json formatted data also has the headers.
>  # Json formatted data doesn't have the column statistics in it.
>  # There is no TestBeeLineDriver test for desc table, thus the actual output 
> is not tested, just some intermediary.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22323) Fix Desc Table bugs

2019-10-10 Thread Miklos Gergely (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miklos Gergely reassigned HIVE-22323:
-


> Fix Desc Table bugs
> ---
>
> Key: HIVE-22323
> URL: https://issues.apache.org/jira/browse/HIVE-22323
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
>
> DESC TABLE operation is having the following bugs:
>  # Whole table descs have two headers.
>  # Table column desc has incorrect long header, while the table is transposed 
> having the headers in the first column.
>  # Json formatted data also has the headers.
>  # Json formatted data doesn't have the column statistics in it.
>  # There is no TestBeeLineDriver test for desc table, thus the actual output 
> is not tested, just some intermediary.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java

2019-10-10 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948585#comment-16948585
 ] 

László Bodor commented on HIVE-21246:
-

+1

> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch, 
> HIVE-21246.2.patch
>
>
> Ultimately, I'd like to get rid of 
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to 
> make it easier to get rid of later.  It's currently buried in 
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods.  If the flag is 
> set to true, the specified SerDe is overwritten and assigned to 
> {{DelimitedJSONSerDe}}.  This is not documented anywhere and it's a weird 
> thing to do, just pass in the required SerDe from the start instead of 
> sending the wrong SerDe and a flag to overwrite it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread mahesh kumar behera (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-22319:
---
Attachment: HIVE-22319.01.patch

> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22319.01.patch
>
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread mahesh kumar behera (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-22319:
---
Status: Patch Available  (was: Open)

> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22319.01.patch
>
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Ivan Suller (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Suller updated HIVE-22322:
---
Status: Patch Available  (was: In Progress)

> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-22322.1.patch
>
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Ivan Suller (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Suller updated HIVE-22322:
---
Attachment: HIVE-22322.1.patch

> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-22322.1.patch
>
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Ivan Suller (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-22322 started by Ivan Suller.
--
> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22322) Remove Netty3 dependency of llap-server

2019-10-10 Thread Ivan Suller (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Suller reassigned HIVE-22322:
--


> Remove Netty3 dependency of llap-server
> ---
>
> Key: HIVE-22322
> URL: https://issues.apache.org/jira/browse/HIVE-22322
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
>
> Llap-server depends on both Netty3 and Netty4. As Netty3 has known security 
> issues fixed only in Netty4 we should eliminate any dependency on Netty3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948567#comment-16948567
 ] 

Hive QA commented on HIVE-22292:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
9s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
44s{color} | {color:red} ql: The patch generated 31 new + 421 unchanged - 1 
fixed = 452 total (was 422) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
12s{color} | {color:red} ql generated 1 new + 1550 unchanged - 0 fixed = 1551 
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Should 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFRank$GenericUDAFHypotheticalSetRankEvaluator$RankAssets
 be a _static_ inner class?  At GenericUDAFRank.java:inner class?  At 
GenericUDAFRank.java:[lines 304-315] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18938/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18938/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18938/yetus/new-findbugs-ql.html
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18938/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18938/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch
>
>
> {code}
>  ::=
>
>
>   
>  ::=
>   RANK
>   | DENSE_RANK
>   | PERCENT_RANK
>   | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL), 
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Marta Kuczora (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-21407:
-
Attachment: HIVE-21407.7.patch

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, 
> HIVE-21407.7.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22321) Setting default nulls last does not take effect when order direction is specified

2019-10-10 Thread Krisztian Kasa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krisztian Kasa updated HIVE-22321:
--
Description: 
{code:java}
SET hive.default.nulls.last=true;
SELECT * FROM t_test ORDER BY col1 ASC;
{code}
{code:java}
POSTHOOK: query: SELECT * FROM t_test ORDER BY col1 ASC
POSTHOOK: type: QUERY
POSTHOOK: Input: default@t_test
 A masked pattern was here 
NULL
NULL
NULL
NULL
3
5
5
{code}
[https://github.com/apache/hive/blob/cb83da943c8919e2ab3751244de5c2879c8fda1d/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g#L2510]

 

cc. [~jcamachorodriguez]

  was:
{code}
SET hive.default.nulls.last=true;
SELECT * FROM t_test ORDER BY col1 ASC;
{code}
{code}
POSTHOOK: query: SELECT * FROM t_test ORDER BY col1 ASC
POSTHOOK: type: QUERY
POSTHOOK: Input: default@t_test
 A masked pattern was here 
NULL
NULL
NULL
NULL
3
5
5
{code}

https://github.com/apache/hive/blob/cb83da943c8919e2ab3751244de5c2879c8fda1d/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g#L2510


> Setting default nulls last does not take effect when order direction is 
> specified
> -
>
> Key: HIVE-22321
> URL: https://issues.apache.org/jira/browse/HIVE-22321
> Project: Hive
>  Issue Type: Bug
>  Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>
> {code:java}
> SET hive.default.nulls.last=true;
> SELECT * FROM t_test ORDER BY col1 ASC;
> {code}
> {code:java}
> POSTHOOK: query: SELECT * FROM t_test ORDER BY col1 ASC
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@t_test
>  A masked pattern was here 
> NULL
> NULL
> NULL
> NULL
> 3
> 5
> 5
> {code}
> [https://github.com/apache/hive/blob/cb83da943c8919e2ab3751244de5c2879c8fda1d/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g#L2510]
>  
> cc. [~jcamachorodriguez]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22321) Setting default nulls last does not take effect when order direction is specified

2019-10-10 Thread Krisztian Kasa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krisztian Kasa reassigned HIVE-22321:
-


> Setting default nulls last does not take effect when order direction is 
> specified
> -
>
> Key: HIVE-22321
> URL: https://issues.apache.org/jira/browse/HIVE-22321
> Project: Hive
>  Issue Type: Bug
>  Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>
> {code}
> SET hive.default.nulls.last=true;
> SELECT * FROM t_test ORDER BY col1 ASC;
> {code}
> {code}
> POSTHOOK: query: SELECT * FROM t_test ORDER BY col1 ASC
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@t_test
>  A masked pattern was here 
> NULL
> NULL
> NULL
> NULL
> 3
> 5
> 5
> {code}
> https://github.com/apache/hive/blob/cb83da943c8919e2ab3751244de5c2879c8fda1d/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g#L2510



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22320:

Description: 
Fs + cluster types are already linked together in MiniClusterType, so it's 
confusing to set them in 2 steps, separately, if it covers an already defined 
minicluster type.
https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
{code}
setClusterType(MiniClusterType.TEZ);
setFsType(QTestMiniClusters.FsType.HDFS);
{code}

> Cluster and fs type settings can be replaced with a single minicluster 
> setting in CliConfigs
> 
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
>
> Fs + cluster types are already linked together in MiniClusterType, so it's 
> confusing to set them in 2 steps, separately, if it covers an already defined 
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22320:

Fix Version/s: 4.0.0

> Cluster and fs type settings can be replaced with a single minicluster 
> setting in CliConfigs
> 
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's 
> confusing to set them in 2 steps, separately, if it covers an already defined 
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22320:

Priority: Minor  (was: Major)

> Cluster and fs type settings can be replaced with a single minicluster 
> setting in CliConfigs
> 
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
>  Issue Type: Bug
>Reporter: László Bodor
>Priority: Minor
> Fix For: 4.0.0
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's 
> confusing to set them in 2 steps, separately, if it covers an already defined 
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22320:

Labels: newbie  (was: )

> Cluster and fs type settings can be replaced with a single minicluster 
> setting in CliConfigs
> 
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Priority: Minor
>  Labels: newbie
> Fix For: 4.0.0
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's 
> confusing to set them in 2 steps, separately, if it covers an already defined 
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs

2019-10-10 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-22320:

Component/s: Testing Infrastructure
 Test

> Cluster and fs type settings can be replaced with a single minicluster 
> setting in CliConfigs
> 
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Priority: Minor
> Fix For: 4.0.0
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's 
> confusing to set them in 2 steps, separately, if it covers an already defined 
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions

2019-10-10 Thread Krisztian Kasa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krisztian Kasa updated HIVE-22292:
--
Attachment: HIVE-22292.1.patch

> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch
>
>
> {code}
>  ::=
>
>
>   
>  ::=
>   RANK
>   | DENSE_RANK
>   | PERCENT_RANK
>   | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL), 
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions

2019-10-10 Thread Krisztian Kasa (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krisztian Kasa updated HIVE-22292:
--
Status: Patch Available  (was: Open)

> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch
>
>
> {code}
>  ::=
>
>
>   
>  ::=
>   RANK
>   | DENSE_RANK
>   | PERCENT_RANK
>   | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL), 
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-21407) Parquet predicate pushdown is not working correctly for char column types

2019-10-10 Thread Marta Kuczora (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marta Kuczora updated HIVE-21407:
-
Attachment: HIVE-21407.6.patch

> Parquet predicate pushdown is not working correctly for char column types
> -
>
> Key: HIVE-21407
> URL: https://issues.apache.org/jira/browse/HIVE-21407
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21407.2.patch, HIVE-21407.3.patch, 
> HIVE-21407.4.patch, HIVE-21407.5.patch, HIVE-21407.6.patch, HIVE-21407.patch
>
>
> If the 'hive.optimize.index.filter' parameter is false, the filter predicate 
> is not pushed to parquet, so the filtering only happens within Hive. If the 
> parameter is true, the filter is pushed to parquet, but for a char type, the 
> value which is pushed to Parquet will be padded with spaces:
> {noformat}
>   @Override
>   public void setValue(String val, int len) {
> super.setValue(HiveBaseChar.getPaddedValue(val, len), -1);
>   }
> {noformat} 
> So if we have a char(10) column which contains the value "apple" and the 
> where condition looks like 'where c='apple'', the value pushed to Paquet will 
> be 'apple' followed by 5 spaces. But the stored values are not padded, so no 
> rows will be returned from Parquet.
> How to reproduce:
> {noformat}
> $ create table ppd (c char(10), v varchar(10), i int) stored as parquet;
> $ insert into ppd values ('apple', 'bee', 1),('apple', 'tree', 2),('hello', 
> 'world', 1),('hello','vilag',3);
> $ set hive.optimize.ppd.storage=true;
> $ set hive.vectorized.execution.enabled=true;
> $ set hive.vectorized.execution.enabled=false;
> $ set hive.optimize.ppd=true;
> $ set hive.optimize.index.filter=true;
> $ set hive.parquet.timestamp.skip.conversion=false;
> $ select * from ppd where c='apple';
> ++++
> | ppd.c  | ppd.v  | ppd.i  |
> ++++
> ++++
> $ set hive.optimize.index.filter=false; or set 
> hive.optimize.ppd.storage=false;
> $ select * from ppd where c='apple';
> +-+++
> |ppd.c| ppd.v  | ppd.i  |
> +-+++
> | apple   | bee| 1  |
> | apple   | tree   | 2  |
> +-+++
> {noformat}
> The issue surfaced after uploading the fix for 
> [HIVE-21327|https://issues.apache.org/jira/browse/HIVE-21327] was uploaded 
> upstream. Before the HIVE-21327 fix, setting the parameter 
> 'hive.parquet.timestamp.skip.conversion' to true in the parquet_ppd_char.q 
> test hid this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22317) Beeline site parser does not handle the variable substitution correctly

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948407#comment-16948407
 ] 

Hive QA commented on HIVE-22317:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982637/HIVE-22317.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 17518 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18936/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18936/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18936/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982637 - PreCommit-HIVE-Build

> Beeline site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>  
>  
>  beeline.hs2.jdbc.url.container
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>  
>  
>  
>  beeline.hs2.jdbc.url.default
>  test
>  
>  
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>  
>  
>  beeline.hs2.jdbc.url.llap
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>  
>  
>  {code}
> beeline fail to connect because it does not parse the substituted value 
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22319) Repl load fails to create partition if the dump is from old version

2019-10-10 Thread mahesh kumar behera (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera reassigned HIVE-22319:
--


> Repl load fails to create partition if the dump is from old version
> ---
>
> Key: HIVE-22319
> URL: https://issues.apache.org/jira/browse/HIVE-22319
> Project: Hive
>  Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>
> The engine field of column  stats in partition descriptor needs to be 
> initialized. Handling needs to be added for column stat events also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22306) Use nonblocking thrift server for metastore

2019-10-10 Thread Qinghui Xu (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948374#comment-16948374
 ] 

Qinghui Xu commented on HIVE-22306:
---

Thanks for the feedback, and that's a very good point.

What I'm currently doing:
 * It turns out that thrift nonblocking server implementations are not 
production ready (there's currently no secure implementations).
 I'm trying first to add SASL support in nonblocking servers, as it will enable 
the use of most common security mechanisms, especially kerberos. Here's the 
work in progress: THRIFT-4889, and there is a PR for it. It would be great if 
hive guys can help to review it so that I can get it merged.
 * I'll try to use the thrift nonblocking sasl server implementation 
(THRIFT-4889) in our hive fork, and test it in our preprod environment 
(eventually go to prod).
 * If everything is doing good, I'll propose a new feature (PR) for hive 
metastore to run with sasl and (optional) nonblocking mode
 * Eventually remove the blocking server for sasl mode, and have only 
nonblocking server.
 * Eventually use nonblocking server for other modes without sasl.

> Use nonblocking thrift server for metastore
> ---
>
> Key: HIVE-22306
> URL: https://issues.apache.org/jira/browse/HIVE-22306
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Qinghui Xu
>Priority: Major
>
> Currently hive metastore's threads are blocking for network io (it's using 
> `TThreadPoolServer` behind the scene), which means with increasing use cases 
> (in our tech stack there are different services relying on it, hiveserver2, 
> spark, presto, and more, all with a significant number of users), to handle 
> all connections it needs either a big thread pool or many instances with 
> smaller thread pools. And often, those metastores will see their thread pool 
> saturated, while the cpu usage is still quite low, just because most 
> connections stay idle and only run a query from time to time. This is thus a 
> great misuse of the computation resources.
> Thus I propose to use a non blocking threading model, and run computation 
> asynchronously. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22317) Beeline site parser does not handle the variable substitution correctly

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948350#comment-16948350
 ] 

Hive QA commented on HIVE-22317:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} beeline in master has 48 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18936/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18936/yetus/patch-asflicense-problems.txt
 |
| modules | C: beeline U: beeline |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18936/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Beeline site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>  
>  
>  beeline.hs2.jdbc.url.container
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>  
>  
>  
>  beeline.hs2.jdbc.url.default
>  test
>  
>  
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>  
>  
>  beeline.hs2.jdbc.url.llap
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>  
>  
>  {code}
> beeline fail to connect because it does not parse the substituted value 
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22239) Scale data size using column value ranges

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948334#comment-16948334
 ] 

Hive QA commented on HIVE-22239:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982636/HIVE-22239.05.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17518 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures
 (batchId=345)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18935/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18935/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18935/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982636 - PreCommit-HIVE-Build

> Scale data size using column value ranges
> -
>
> Key: HIVE-22239
> URL: https://issues.apache.org/jira/browse/HIVE-22239
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, 
> HIVE-22239.03.patch, HIVE-22239.04.patch, HIVE-22239.04.patch, 
> HIVE-22239.05.patch, HIVE-22239.patch
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Currently, min/max values for columns are only used to determine whether a 
> certain range filter falls out of range and thus filters all rows or none at 
> all. If it does not, we just use a heuristic that the condition will filter 
> 1/3 of the input rows. Instead of using that heuristic, we can use another 
> one that assumes that data will be uniformly distributed across that range, 
> and calculate the selectivity for the condition accordingly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=326191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326191
 ]

ASF GitHub Bot logged work on HIVE-19261:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 08:59
Start Date: 10/Oct/19 08:59
Worklog Time Spent: 10m 
  Work Description: dlavati commented on pull request #807: HIVE-19261: 
Avro SerDe's InstanceCache should not be synchronized on retrieve
URL: https://github.com/apache/hive/pull/807#discussion_r333403943
 
 

 ##
 File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
 ##
 @@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws 
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
-  public synchronized Instance retrieve(SeedObject hv,
-  Set seenSchemas) throws AvroSerdeException {
+  public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
 if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
 
 if(cache.containsKey(hv)) {
   if(LOG.isDebugEnabled()) LOG.debug("Returning cache result.");
   return cache.get(hv);
+} else {
+  if(LOG.isDebugEnabled()) LOG.debug("Creating new instance and storing in 
cache");
 
 Review comment:
   @xhumanoid thanks for the contribution! I don't have committer rights to 
approve your patch, but LGTM with a few comments:
   - a small nitpick, but AFAIK, `isDebugEnabled` doesn't provide benefits 
here, only if you'd append/toString an expensive operation's result (yes, this 
is also ignored in a lot of places )
   - to approve (+1) your changes, you'll have to become the assignee of the 
jira ticket that's currently under Fangshi Li (if that's fine with him) and 
change it's state with `Submit a patch` with an attached patch file (the 
previously attached file didn't get checked, as the issue state didn't change)
   - once all tests pass for it (might require a few reattachments/reruns), 
then someone will be able to approve it (also see 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326191)
Time Spent: 40m  (was: 0.5h)

> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
>  Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Fangshi Li
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in 
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache 
> synchronized. While it should make InstanceCache thread-safe, making retrieve 
> synchronized for the cache can be expensive in highly concurrent environment 
> like Spark, as multiple threads need to be synchronized on entering the 
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the 
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use 
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the 
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still 
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future, 
> we should move to computeIfAbsent when Hive requires java 8.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=326190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326190
 ]

ASF GitHub Bot logged work on HIVE-19261:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 08:57
Start Date: 10/Oct/19 08:57
Worklog Time Spent: 10m 
  Work Description: dlavati commented on pull request #807: HIVE-19261: 
Avro SerDe's InstanceCache should not be synchronized on retrieve
URL: https://github.com/apache/hive/pull/807#discussion_r333403943
 
 

 ##
 File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
 ##
 @@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws 
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
-  public synchronized Instance retrieve(SeedObject hv,
-  Set seenSchemas) throws AvroSerdeException {
+  public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
 if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
 
 if(cache.containsKey(hv)) {
   if(LOG.isDebugEnabled()) LOG.debug("Returning cache result.");
   return cache.get(hv);
+} else {
+  if(LOG.isDebugEnabled()) LOG.debug("Creating new instance and storing in 
cache");
 
 Review comment:
   @xhumanoid thanks for the contribution! I don't have committer rights to 
approve your patch, but LGTM with a few comments:
   - a small nitpick, but AFAIK, `isDebugEnabled` doesn't provide benefits 
here, only if you'd append/toString an expensive operation's result
   - to approve (+1) your changes, you'll have to become the assignee of the 
jira ticket that's currently under Fangshi Li (if that's fine with him) and 
change it's state with `Submit a patch` with an attached patch file (the 
previously attached file didn't get checked, as the issue state didn't change)
   - once all tests pass for it (might require a few reattachments/reruns), 
then someone will be able to approve it (also see 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326190)
Time Spent: 0.5h  (was: 20m)

> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
>  Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Fangshi Li
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in 
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache 
> synchronized. While it should make InstanceCache thread-safe, making retrieve 
> synchronized for the cache can be expensive in highly concurrent environment 
> like Spark, as multiple threads need to be synchronized on entering the 
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the 
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use 
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the 
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still 
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future, 
> we should move to computeIfAbsent when Hive requires java 8.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22239) Scale data size using column value ranges

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948308#comment-16948308
 ] 

Hive QA commented on HIVE-22239:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
11s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
13s{color} | {color:red} ql generated 1 new + 1549 unchanged - 1 fixed = 1550 
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to uniformWithinRange in 
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.process(Node,
 Stack, NodeProcessorCtx, Object[])  At 
StatsRulesProcFactory.java:org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$JoinStatsRule.process(Node,
 Stack, NodeProcessorCtx, Object[])  At StatsRulesProcFactory.java:[line 2025] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18935/dev-support/hive-personality.sh
 |
| git revision | master / cb83da9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18935/yetus/new-findbugs-ql.html
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18935/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18935/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Scale data size using column value ranges
> -
>
> Key: HIVE-22239
> URL: https://issues.apache.org/jira/browse/HIVE-22239
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, 
> HIVE-22239.03.patch,

[jira] [Commented] (HIVE-22238) PK/FK selectivity estimation underscales estimations

2019-10-10 Thread Zoltan Haindrich (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948296#comment-16948296
 ] 

Zoltan Haindrich commented on HIVE-22238:
-

I went after this - but forgot to write an update here...so what's happening is 
somewhat both, now I think that the rescaling is accurate and I agree with the 
logic...but when calcite pushes the filter predicates to the other branch as 
well it ends up downscaling by the same factor again - hence my patch have 
solved some case...I'll try to get back to this sooner than later :)
my current idea is to somehow identify that the FK column in question is not 
filtered so far - so that we may downscale it by the PK factor

> PK/FK selectivity estimation underscales estimations
> 
>
> Key: HIVE-22238
> URL: https://issues.apache.org/jira/browse/HIVE-22238
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22238.01.patch
>
>
> at [this 
> point|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2182]
>  the parent operators rownum is scaled according to pkfkselectivity
> however [pkfkselectivity is 
> computed|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2157]
>  on a whole subtree.
> Scaling it by that amount will count in estimation already used when 
> parentstats was calculated...so depending on the number of upstream joins - 
> this may lead to severe underestimations



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=326158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326158
 ]

ASF GitHub Bot logged work on HIVE-19261:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 07:50
Start Date: 10/Oct/19 07:50
Worklog Time Spent: 10m 
  Work Description: xhumanoid commented on issue #807: HIVE-19261: Avro 
SerDe's InstanceCache should not be synchronized on retrieve
URL: https://github.com/apache/hive/pull/807#issuecomment-540443922
 
 
   @kgyrtkirk @dlavati could you review?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326158)
Time Spent: 20m  (was: 10m)

> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
>  Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Fangshi Li
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in 
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache 
> synchronized. While it should make InstanceCache thread-safe, making retrieve 
> synchronized for the cache can be expensive in highly concurrent environment 
> like Spark, as multiple threads need to be synchronized on entering the 
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the 
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use 
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the 
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still 
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future, 
> we should move to computeIfAbsent when Hive requires java 8.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22303) TestObjectStore starts some deadline timers which are never stopped

2019-10-10 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-22303:

Fix Version/s: 4.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

pushed to master. Thank you Laszlo for reviewing the changes!

> TestObjectStore starts some deadline timers which are never stopped
> ---
>
> Key: HIVE-22303
> URL: https://issues.apache.org/jira/browse/HIVE-22303
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22303.01.patch, HIVE-22303.01.patch
>
>
> because these timers are not stopped; they may stay there as a threadlocal; 
> and eventually time out since the disarm logic is missing...
> https://github.com/apache/hive/blob/d907dfe68ed84714d62a22e5191efa616eab2b24/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java#L373



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948293#comment-16948293
 ] 

Hive QA commented on HIVE-14302:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982627/HIVE-14302.5.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17520 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking7 
(batchId=353)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/18934/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18934/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18934/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12982627 - PreCommit-HIVE-Build

> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14302.2.patch, HIVE-14302.3.patch, 
> HIVE-14302.4.patch, HIVE-14302.5.patch, HIVE-14302.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the 
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because 
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK  
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
>   TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
>   predicate: (a is not null and true) (type: boolean)
>   Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
>   key expressions: _col0 (type: decimal(11,2))
>   sort order: +
>   Map-reduce partition columns: _col0 (type: decimal(11,2))
>   Join Operator (JOIN_8)
> condition map:
>  Inner Join 0 to 1
> keys:
>   0 _col0 (type: decimal(11,2))
>   1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
>   aggregations: count(1)
>   mode: hash
>   outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the 
> join to be able to compare HiveDecimal as-is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21866) LLAP status service driver may get stuck with wrong Yarn app ID

2019-10-10 Thread Sean Roberts (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948289#comment-16948289
 ] 

Sean Roberts commented on HIVE-21866:
-

Workaround for anyone that hits this and doesn't have the fixed version of Hive:
```
hdfs dfs -rm /user/hive/.yarn/services/llap0/llap0.json
```

> LLAP status service driver may get stuck with wrong Yarn app ID
> ---
>
> Key: HIVE-21866
> URL: https://issues.apache.org/jira/browse/HIVE-21866
> Project: Hive
>  Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21866.0.patch
>
>
> LLAPStatusDriver might get stuck polling status from Yarn if the following 
> happen in this order:
>  * there was a running LLAP Yarn app previously which is now finished / killed
>  * Yarn was restarted
>  * LLAPStatusDriver is invoked before any new LLAP app gets kicked off
>  * LLAPStatusDriver receives the old app ID, which is then cached in the Yarn 
> serviceClient object (no evicition)
>  * In the meantime if any new LLAP app gets kicked off, LLAPStatusDriver will 
> not see it, as it constantly retries fetching info about the wrong, old app 
> ID (this is because we don't create new serviceClient objects)
> {code:java}
> ERROR status.LlapStatusServiceDriver: FAILED: 20: Failed to get Yarn AppReport
> org.apache.hadoop.hive.llap.cli.status.LlapStatusCliException: 20: Failed to 
> get Yarn AppReport
>   at 
> org.apache.hadoop.hive.llap.cli.status.LlapStatusServiceDriver.getAppReport(LlapStatusServiceDriver.java:292)
>  [hive-llap-server-3.1.0.7.0.0.0-112.jar:3.1.0.7.0.0.0-134]
>   at 
> org.apache.hadoop.hive.llap.cli.status.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:209)
>  [hive-llap-server-3.1.0.7.0.0.0-112.jar:3.1.0.7.0.0.0-134]
>   at 
> org.apache.hadoop.hive.llap.cli.status.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:537)
>  [hive-llap-server-3.1.0.7.0.0.0-112.jar:3.1.0.7.0.0.0-134]{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948262#comment-16948262
 ] 

Hive QA commented on HIVE-14302:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
17s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18934/dev-support/hive-personality.sh
 |
| git revision | master / 37a0675 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18934/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-18934/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14302.2.patch, HIVE-14302.3.patch, 
> HIVE-14302.4.patch, HIVE-14302.5.patch, HIVE-14302.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the 
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because 
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK  
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
>   TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
>

[jira] [Updated] (HIVE-22317) Beeline site parser does not handle the variable substitution correctly

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-22317:
--
Attachment: HIVE-22317.patch
Status: Patch Available  (was: Open)

> Beeline site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>  
>  
>  beeline.hs2.jdbc.url.container
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>  
>  
>  
>  beeline.hs2.jdbc.url.default
>  test
>  
>  
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>  
>  
>  beeline.hs2.jdbc.url.llap
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>  
>  
>  {code}
> beeline fail to connect because it does not parse the substituted value 
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22317) Beeline site parser does not handle the variable substitution correctly

2019-10-10 Thread Rajkumar Singh (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh reassigned HIVE-22317:
-


> Beeline site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>  
>  
>  beeline.hs2.jdbc.url.container
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>  
>  
>  
>  beeline.hs2.jdbc.url.default
>  test
>  
>  
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>  
>  
>  beeline.hs2.jdbc.url.llap
>  
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>  
>  
>  {code}
> beeline fail to connect because it does not parse the substituted value 
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url: 
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-22274) Upgrade Calcite version to 1.21.0

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948245#comment-16948245
 ] 

Hive QA commented on HIVE-22274:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982626/HIVE-22274.5.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 79 failed/errored test(s), 17361 tests 
executed
*Failed tests:*
{noformat}
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestChainFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestCustomQueryFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestGroupFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestHiveMetaToolCommandLine - did not produce a TEST-*.xml file (likely timed 
out) (batchId=227)
TestLdapSearch - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestLdapUtils - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestMetaStoreConnectionUrlHook - did not produce a TEST-*.xml file (likely 
timed out) (batchId=227)
TestMetaToolTaskExecuteJDOQLQuery - did not produce a TEST-*.xml file (likely 
timed out) (batchId=227)
TestMetaToolTaskListFSRoot - did not produce a TEST-*.xml file (likely timed 
out) (batchId=227)
TestMetaToolTaskUpdateLocation - did not produce a TEST-*.xml file (likely 
timed out) (batchId=227)
TestMetastoreSchemaTool - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestMsckCheckPartitions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestQueryFactory - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestRemoteHiveMetaStoreZK - did not produce a TEST-*.xml file (likely timed 
out) (batchId=227)
TestSchemaToolForMetastore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=227)
TestSchemaToolTaskDrop - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestSearchResultHandler - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestUserFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
TestUserSearchFilter - did not produce a TEST-*.xml file (likely timed out) 
(batchId=227)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=291)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2] 
(batchId=299)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select] 
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog3] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_union] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_case] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when] 
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_when] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_const_type] 
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input9] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin2] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_10] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optimize_filter_literal] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_boolexpr] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udf_col] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[remove_exprs_stats] 
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] 
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4] 
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6] 
(batchId=46)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_key_range]
 (batchId=106)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_pushdown] 
(batchId=106)

[jira] [Commented] (HIVE-22274) Upgrade Calcite version to 1.21.0

2019-10-10 Thread Hive QA (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948234#comment-16948234
 ] 

Hive QA commented on HIVE-22274:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
12s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m  
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
46s{color} | {color:red} ql: The patch generated 31 new + 345 unchanged - 11 
fixed = 376 total (was 356) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  
5s{color} | {color:red} root: The patch generated 31 new + 345 unchanged - 11 
fixed = 376 total (was 356) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
16s{color} | {color:red} ql generated 9 new + 1547 unchanged - 3 fixed = 1556 
total (was 1550) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
2s{color} | {color:red} ql generated 4 new + 96 unchanged - 4 fixed = 100 total 
(was 100) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  6m 
41s{color} | {color:red} root generated 4 new + 333 unchanged - 4 fixed = 337 
total (was 337) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
16s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m  1s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to joinInfo in 
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
 RelNode, RexNode, Set, JoinRelType, boolean)  At 
HiveRelFactories.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
 RelNode, RexNode, Set, JoinRelType, boolean)  At HiveRelFactories.java:[line 
171] |
|  |  Dead store to joinInfo in 
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
 RelNode, RexNode, JoinRelType, Set, boolean)  At 
HiveRelFactories.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
 RelNode, RexNode, JoinRelType, Set, boolean)  At HiveRelFactories.java:[line 
158] |
|  |  Dead store to joinInfo in 
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveSemiJoinFactoryImpl.createSemiJoin(RelNode,
 RelNode, RexNode)  At 
HiveRelFactories.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveSemiJoinFactoryImpl.createSemiJoin(RelNode,
 RelNode, RexNode)  At HiveRelFactories.java:[line 188] |
|  |  Dead store to rightKeys in 
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(LogicalCorrelate)
  At

[jira] [Work logged] (HIVE-22239) Scale data size using column value ranges

2019-10-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-22239?focusedWorklogId=326131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326131
 ]

ASF GitHub Bot logged work on HIVE-22239:
-

Author: ASF GitHub Bot
Created on: 10/Oct/19 06:00
Start Date: 10/Oct/19 06:00
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on issue #787: HIVE-22239
URL: https://github.com/apache/hive/pull/787#issuecomment-540389710
 
 
   @kgyrtkirk , I pushed a new commit containing only the range change and 
addressing your comments ; I will upload a follow-up for the timestamp column 
stats propagation in a new PR. Can you take another look?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 326131)
Time Spent: 4.5h  (was: 4h 20m)

> Scale data size using column value ranges
> -
>
> Key: HIVE-22239
> URL: https://issues.apache.org/jira/browse/HIVE-22239
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, 
> HIVE-22239.03.patch, HIVE-22239.04.patch, HIVE-22239.04.patch, 
> HIVE-22239.05.patch, HIVE-22239.patch
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Currently, min/max values for columns are only used to determine whether a 
> certain range filter falls out of range and thus filters all rows or none at 
> all. If it does not, we just use a heuristic that the condition will filter 
> 1/3 of the input rows. Instead of using that heuristic, we can use another 
> one that assumes that data will be uniformly distributed across that range, 
> and calculate the selectivity for the condition accordingly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

97 matches

Mail list logo