date:20180807

[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Alice Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alice Fan updated HIVE-20246:
-
Status: Open  (was: Patch Available)

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch, HIVE-20246.2.patch, 
> HIVE-20246.3.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Alice Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.3.patch
Status: Patch Available  (was: Open)

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch, HIVE-20246.2.patch, 
> HIVE-20246.3.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572703#comment-16572703
 ] 

Hive QA commented on HIVE-20246:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 6s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 5 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
12s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m  3s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13096/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13096/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13096/yetus/whitespace-tabs.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13096/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
 |
| modules | C: standalone-metastore/metastore-server U: 
standalone-metastore/metastore-server |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13096/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch, HIVE-20246.2.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');'

[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572697#comment-16572697
 ] 

Hive QA commented on HIVE-20315:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} serde in master has 195 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
53s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} serde: The patch generated 2 new + 6 unchanged - 0 
fixed = 8 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
21s{color} | {color:red} root: The patch generated 2 new + 9 unchanged - 0 
fixed = 11 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m  
9s{color} | {color:red} ql: The patch generated 88 new + 2191 unchanged - 78 
fixed = 2279 total (was 2269) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
2s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  9m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13095/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13095/yetus/diff-checkstyle-serde.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13095/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13095/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13095/yetus/whitespace-eol.txt
 |
| modules | C: serde . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13095/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>

[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572692#comment-16572692
 ] 

Hive QA commented on HIVE-20315:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934742/HIVE-20315.08.patch

{color:green}SUCCESS:{color} +1 due to 119 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14869 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal]
 (batchId=170)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13095/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13095/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13095/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934742 - PreCommit-HIVE-Build

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch, 
> HIVE-20315.09.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Attachment: HIVE-20315.09.patch

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch, 
> HIVE-20315.09.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Status: Patch Available  (was: In Progress)

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch, 
> HIVE-20315.09.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Status: In Progress  (was: Patch Available)

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Alice Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.2.patch
Status: Patch Available  (was: Open)

Improving style.

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch, HIVE-20246.2.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Alice Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alice Fan updated HIVE-20246:
-
Status: Open  (was: Patch Available)

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Teddy Choi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572659#comment-16572659
 ] 

Teddy Choi commented on HIVE-20312:
---

+1 Looks good to me.

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572650#comment-16572650
 ] 

Hive QA commented on HIVE-20246:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934740/HIVE-20246.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14869 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13094/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13094/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13094/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934740 - PreCommit-HIVE-Build

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Eric Wohlstadter (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572630#comment-16572630
 ] 

Eric Wohlstadter commented on HIVE-20312:
-

I modified {{LlapArrowRowInputFormat}} so this will be tested by the existing 
{{TestJdbcWithMiniLlapArrow}}

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Teddy Choi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572612#comment-16572612
 ] 

Teddy Choi edited comment on HIVE-20315 at 8/8/18 3:44 AM:
---

+1 Looks good to me. I ran convert_decimal64_to_decimal.q test with 
HIVE-20315.08.patch patch on my machine. It had few differences. Other things 
look good.


was (Author: teddy.choi):
+1 Looks good to me. I ran convert_decimal64_to_decimal.q test with 
HIVE-20315.08.patch patch on my machine. It had just one difference as 
following. Other things look good.

LINE 184: bigTableKeyExpressions: ConvertDecimal64ToDecimal(col 
3:decimal(9,2)/DECIMAL_64) -> 20:decimal(9,2)

Should be: bigTableKeyExpressions: ConvertDecimal64ToDecimal(col 
3:decimal(9,2)/DECIMAL_64) -> 21:decimal(9,2)

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Teddy Choi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572612#comment-16572612
 ] 

Teddy Choi edited comment on HIVE-20315 at 8/8/18 3:37 AM:
---

+1 Looks good to me. I ran convert_decimal64_to_decimal.q test with 
HIVE-20315.08.patch patch on my machine. It had just one difference as 
following. Other things look good.

LINE 184: bigTableKeyExpressions: ConvertDecimal64ToDecimal(col 
3:decimal(9,2)/DECIMAL_64) -> 20:decimal(9,2)

Should be: bigTableKeyExpressions: ConvertDecimal64ToDecimal(col 
3:decimal(9,2)/DECIMAL_64) -> 21:decimal(9,2)


was (Author: teddy.choi):
+1 Looks good to me. I ran convert_decimal64_to_decimal.q test with 
HIVE-20315.08.patch patch on my machine and it succeeded.

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572617#comment-16572617
 ] 

Hive QA commented on HIVE-20246:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 6s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
13s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 8 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13094/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13094/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13094/yetus/whitespace-tabs.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13094/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
 |
| modules | C: standalone-metastore/metastore-server U: 
standalone-metastore/metastore-server |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13094/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not

[jira] [Commented] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Teddy Choi (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572612#comment-16572612
 ] 

Teddy Choi commented on HIVE-20315:
---

+1 Looks good to me. I ran convert_decimal64_to_decimal.q test with 
HIVE-20315.08.patch patch on my machine and it succeeded.

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572606#comment-16572606
 ] 

Hive QA commented on HIVE-20255:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934736/HIVE-20255.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14868 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13093/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13093/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13093/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934736 - PreCommit-HIVE-Build

> Review LevelOrderWalker.java
> 
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch, 
> HIVE-20255.3.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
>   if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some 
> real work, so do not throw away the work after checking for null.  Simply 
> call once and store the results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572581#comment-16572581
 ] 

Hive QA commented on HIVE-20255:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
42s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 2 
fixed = 1 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13093/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13093/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review LevelOrderWalker.java
> 
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch, 
> HIVE-20255.3.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
>   if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some 
> real work, so do not throw away the work after checking for null.  Simply 
> call once and store the results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572566#comment-16572566
 ] 

Hive QA commented on HIVE-20312:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934723/HIVE-20312.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14868 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13092/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13092/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13092/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934723 - PreCommit-HIVE-Build

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Attachment: HIVE-20315.08.patch

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Status: Patch Available  (was: In Progress)

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch, HIVE-20315.08.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20315) Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary casts/conversions

2018-08-07 Thread Matt McCline (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-20315:

Status: In Progress  (was: Patch Available)

> Vectorization: Fix more NULL / Wrong Results issues and avoid unnecessary 
> casts/conversions
> ---
>
> Key: HIVE-20315
> URL: https://issues.apache.org/jira/browse/HIVE-20315
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20315.01.patch, HIVE-20315.02.patch, 
> HIVE-20315.03.patch, HIVE-20315.04.patch, HIVE-20315.05.patch, 
> HIVE-20315.06.patch, HIVE-20315.07.patch
>
>
> Generate multi-byte Unicode characters in addition to regular single byte 
> characters for random data.
> Don't CAST from STRING/VARCHAR/CHAR TO STRING since all are stored in 
> vectorization without padding.
> Fix vectorized BETWEEN expression work to avoid unnecessary CAST of DECIMAL 
> constants.
> Fix NULL / Wrong Results issues in VectorElt.
> Change performance Q files to generate non-user EXPLAIN with VECTORIZATION 
> display so unnecesary CAST / DECIMAL_64 conversions are visible.
> Add test for vectorized STRUCT field reference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572533#comment-16572533
 ] 

Hive QA commented on HIVE-20312:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} llap-ext-client: The patch generated 1 new + 42 
unchanged - 0 fixed = 43 total (was 42) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13092/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13092/yetus/diff-checkstyle-llap-ext-client.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13092/yetus/whitespace-eol.txt
 |
| modules | C: llap-ext-client U: llap-ext-client |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13092/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property

2018-08-07 Thread Alice Fan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alice Fan updated HIVE-20246:
-
Attachment: HIVE-20246.1.patch
Status: Patch Available  (was: Open)

> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.1.patch
>
>
> By default, Hive collects stats when running operations like alter table 
> partition(s), create table, and create external table. However, collecting 
> stats requires Metastore lists all files under the table directory and the 
> file listing operation can be very expensive particularly on filesystems like 
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively 
> prevent stats collection. 
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the 
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be 
> selectively prevent stats collection when doing alter table partition(s) 
> operation at table level. For example, set 'Alter Table S3_Table set 
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect 
> stats for the specified S3_Table when alter table add partition(key1=val1, 
> key2=val2);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572526#comment-16572526
 ] 

Hive QA commented on HIVE-20044:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934722/HIVE-20044.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14869 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13091/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13091/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13091/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934722 - PreCommit-HIVE-Build

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572516#comment-16572516
 ] 

Hive QA commented on HIVE-20044:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
23s{color} | {color:blue} storage-api in master has 48 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
45s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 46 new + 225 unchanged - 40 
fixed = 271 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13091/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13091/yetus/diff-checkstyle-ql.txt
 |
| modules | C: storage-api ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13091/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic

2018-08-07 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572504#comment-16572504
 ] 

Eugene Koifman commented on HIVE-19800:
---

todo: BUG-107516

> Create separate submodules for pre and post upgrade and add rename file logic
> -
>
> Key: HIVE-19800
> URL: https://issues.apache.org/jira/browse/HIVE-19800
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, 
> HIVE-19800.03.patch, HIVE-19800.04.patch
>
>
> this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't 
> landed yet
> this includes file rename logic and HIVE-19750 since it hasn't landed yet 
> either
>  
> cc [~jdere]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20225) SerDe to support Teradata Binary Format

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572495#comment-16572495
 ] 

Hive QA commented on HIVE-20225:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934720/HIVE-20225.8.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14883 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13090/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13090/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13090/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934720 - PreCommit-HIVE-Build

> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
>  Issue Type: New Feature
>  Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch, 
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, 
> HIVE-20225.6.patch, HIVE-20225.7.patch, HIVE-20225.8.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will 
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive 
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT. 
> However, the Hive could not directly utilize/generate these binary format 
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata 
> Binary Format file transparently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Ashutosh Chauhan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572493#comment-16572493
 ] 

Ashutosh Chauhan commented on HIVE-20279:
-

OK. [~nishantbangarwa] Suggestion is to make getPos() in DruidRecordReader to 
return -1; (instead of 0) so that we never get in this loop.

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Ashutosh Chauhan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572492#comment-16572492
 ] 

Ashutosh Chauhan commented on HIVE-20335:
-

+1

> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20335.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java

2018-08-07 Thread BELUGA BEHR (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-20255:
---
Status: Patch Available  (was: Open)

Fixed check-style

> Review LevelOrderWalker.java
> 
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch, 
> HIVE-20255.3.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
>   if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some 
> real work, so do not throw away the work after checking for null.  Simply 
> call once and store the results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java

2018-08-07 Thread BELUGA BEHR (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-20255:
---
Attachment: HIVE-20255.3.patch

> Review LevelOrderWalker.java
> 
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch, 
> HIVE-20255.3.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
>   if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some 
> real work, so do not throw away the work after checking for null.  Simply 
> call once and store the results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java

2018-08-07 Thread BELUGA BEHR (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-20255:
---
Status: Open  (was: Patch Available)

> Review LevelOrderWalker.java
> 
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.2.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
>   if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some 
> real work, so do not throw away the work after checking for null.  Simply 
> call once and store the results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20225) SerDe to support Teradata Binary Format

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572477#comment-16572477
 ] 

Hive QA commented on HIVE-20225:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} serde in master has 195 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} serde: The patch generated 4 new + 0 unchanged - 0 
fixed = 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
35s{color} | {color:red} ql: The patch generated 1 new + 0 unchanged - 0 fixed 
= 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
46s{color} | {color:red} serde generated 1 new + 195 unchanged - 0 fixed = 196 
total (was 195) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
|  |  Format-string method String.format(String, Object[]) called with format 
string "The byte num %s of HiveCharWritable is more than the byte num %s we can 
hold. The content of HiveCharWritable is " wants 2 arguments but is given 3 in 
org.apache.hadoop.hive.serde2.teradata.TeradataBinaryDataOutputStream.writeChar(HiveCharWritable,
 int)  At TeradataBinaryDataOutputStream.java:with format string "The byte num 
%s of HiveCharWritable is more than the byte num %s we can hold. The content of 
HiveCharWritable is " wants 2 arguments but is given 3 in 
org.apache.hadoop.hive.serde2.teradata.TeradataBinaryDataOutputStream.writeChar(HiveCharWritable,
 int)  At TeradataBinaryDataOutputStream.java:[line 187] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13090/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13090/yetus/diff-checkstyle-serde.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13090/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13090/yetus/new-findbugs-serde.html
 |
| modules | C: serde ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13090/yetus.txt |
|

[jira] [Commented] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Matt McCline (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572470#comment-16572470
 ] 

Matt McCline commented on HIVE-20044:
-

Ok, I looked at the newest changes (patch #3) and it looks good to me. +1

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572457#comment-16572457
 ] 

Hive QA commented on HIVE-20335:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934709/HIVE-20335.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14869 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13089/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13089/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13089/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934709 - PreCommit-HIVE-Build

> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20335.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572401#comment-16572401
 ] 

Hive QA commented on HIVE-20335:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  2m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13089/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13089/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20335.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572395#comment-16572395
 ] 

Hive QA commented on HIVE-20279:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934697/HIVE-20279.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14868 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13087/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13087/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13087/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934697 - PreCommit-HIVE-Build

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20312:

Status: Patch Available  (was: Open)

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20312:

Attachment: HIVE-20312.3.patch

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, 
> HIVE-20312.3.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20312:

Status: Open  (was: Patch Available)

> Allow arrow clients to use their own BufferAllocator with 
> LlapOutputFormatService
> -
>
> Key: HIVE-20312
> URL: https://issues.apache.org/jira/browse/HIVE-20312
> Project: Hive
>  Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch
>
>
> Clients should be able to provide their own BufferAllocator to 
> LlapBaseInputFormat if allocator operations depend on client-side logic. For 
> example, clients may want to manage the allocator hierarchy per client-side 
> task, thread, etc.. 
> Currently the client is forced to use one global RootAllocator per process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20044:

Attachment: HIVE-20044.3.patch

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Eric Wohlstadter
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20044:

Status: Patch Available  (was: Open)

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Eric Wohlstadter
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572363#comment-16572363
 ] 

Eric Wohlstadter commented on HIVE-20044:
-

[~mmccline] [~teddy.choi]

Updated the patch to merge with changes from HIVE-20300

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter reassigned HIVE-20044:
---

Assignee: Teddy Choi  (was: Eric Wohlstadter)

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.3.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-20044:

Status: Open  (was: Patch Available)

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20044) Arrow Serde should pad char values and handle empty strings correctly

2018-08-07 Thread Eric Wohlstadter (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter reassigned HIVE-20044:
---

Assignee: Eric Wohlstadter  (was: Teddy Choi)

> Arrow Serde should pad char values and handle empty strings correctly
> -
>
> Key: HIVE-20044
> URL: https://issues.apache.org/jira/browse/HIVE-20044
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Eric Wohlstadter
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-20044.1.branch-3.patch, HIVE-20044.1.patch, 
> HIVE-20044.1.patch, HIVE-20044.2.patch, HIVE-20044.patch
>
>
> When Arrow Serde serializes char values, it loses padding. Also when it 
> counts empty strings, sometimes it makes a smaller number. It should pad char 
> values and handle empty strings correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572349#comment-16572349
 ] 

Hive QA commented on HIVE-20279:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
48s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13087/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13087/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format

2018-08-07 Thread Lu Li (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lu Li updated HIVE-20225:
-
Status: Patch Available  (was: Open)

> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
>  Issue Type: New Feature
>  Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch, 
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, 
> HIVE-20225.6.patch, HIVE-20225.7.patch, HIVE-20225.8.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will 
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive 
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT. 
> However, the Hive could not directly utilize/generate these binary format 
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata 
> Binary Format file transparently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20225) SerDe to support Teradata Binary Format

2018-08-07 Thread Lu Li (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lu Li updated HIVE-20225:
-
Attachment: HIVE-20225.8.patch

> SerDe to support Teradata Binary Format
> ---
>
> Key: HIVE-20225
> URL: https://issues.apache.org/jira/browse/HIVE-20225
> Project: Hive
>  Issue Type: New Feature
>  Components: Serializers/Deserializers
>Reporter: Lu Li
>Assignee: Lu Li
>Priority: Major
> Attachments: HIVE-20225.1.patch, HIVE-20225.2.patch, 
> HIVE-20225.3.patch, HIVE-20225.4.patch, HIVE-20225.5-branch-2.patch, 
> HIVE-20225.6.patch, HIVE-20225.7.patch, HIVE-20225.8.patch
>
>
> When using TPT/BTEQ to export/import Data from Teradata, Teradata will 
> generate/require binary files based on the schema.
> A Customized SerDe is needed in order to directly read these files from Hive 
> or write these files in order to load back to TD.
> {code:java}
> CREATE EXTERNAL TABLE `TABLE1`(
> ...)
> PARTITIONED BY (
> ...)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.contrib.serde2.TeradataBinarySerde'
> STORED AS INPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileInputFormat'
> OUTPUTFORMAT
>  
> 'org.apache.hadoop.hive.contrib.fileformat.teradata.TeradataBinaryFileOutputFormat'
> LOCATION ...;
> SELECT * FROM `TABLE1`;{code}
> Problem Statement:
> Right now the fast way to export/import data from Teradata is using TPT. 
> However, the Hive could not directly utilize/generate these binary format 
> because it doesn't have a SerDe for these files.
> Result:
> Provided with the SerDe, Hive can operate upon/generate the exported Teradata 
> Binary Format file transparently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Gopal V (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572326#comment-16572326
 ] 

Gopal V commented on HIVE-20279:


No, headers are used for other binary sensor formats (footers only work for 
TextInputFormat).

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572298#comment-16572298
 ] 

Hive QA commented on HIVE-20332:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934695/HIVE-20332.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 192 failed/errored test(s), 14868 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats2] 
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats] 
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
 (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4] 
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown] 
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join42] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join47] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_alt_syntax] 
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_2] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_4] 
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual2]
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual4]
 (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_parse] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[leftsemijoin] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_mapjoin] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoins] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] 
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6] 
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_context] 
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2] 
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_6]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer3]
 (batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer6]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_3]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_4]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
 (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[leftsemijoin]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[limit_join_transpose]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_mapjoin]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_5]
 (batchId=162)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Ashutosh Chauhan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572295#comment-16572295
 ] 

Ashutosh Chauhan commented on HIVE-20279:
-

Isnt header/footer only relevant for TextInputFormat ?

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Gopal V (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572272#comment-16572272
 ] 

Gopal V edited comment on HIVE-20279 at 8/7/18 8:23 PM:


[~ashutoshc]: this patch is probably too fragile - I chased this bug down to 
DruidQueryRecordReader not implementing ::getPos().

https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidQueryRecordReader.java#L145


was (Author: gopalv):
[~ashutoshc]: this patch is probably too fragile - I chased this bug down to 
DruidQueryRecordReader not implementing ::getPos().

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Gopal V (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572272#comment-16572272
 ] 

Gopal V commented on HIVE-20279:


[~ashutoshc]: this patch is probably too fragile - I chased this bug down to 
DruidQueryRecordReader not implementing ::getPos().

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572240#comment-16572240
 ] 

Hive QA commented on HIVE-20332:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
29s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
44s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 4 new + 213 unchanged - 1 
fixed = 217 total (was 214) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13086/dev-support/hive-personality.sh
 |
| git revision | master / c0f63bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13086/yetus/diff-checkstyle-ql.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13086/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20332.patch
>
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer 
> (this should be fixed by HIVE-20313). Even if we did, we always assume 
> uniform distribution of the column values, which can easily lead to 
> overestimations on the number of rows read when we filter on 
> {{ROW\_\_ID.writeId}} for materialized views (think about a large transaction 
> for MV creation and then small ones for incremental maintenance). This 
> overestimation can lead to incremental view maintenance not being triggered 
>

[jira] [Commented] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Ashutosh Chauhan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572239#comment-16572239
 ] 

Ashutosh Chauhan commented on HIVE-20279:
-

+1

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20335:
---
Attachment: HIVE-20335.patch

> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20335.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20335:
---
Status: Patch Available  (was: In Progress)

> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-20335 started by Jesus Camacho Rodriguez.
--
> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20335) Add tests for materialized view rewriting with composite aggregation functions

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-20335:
--


> Add tests for materialized view rewriting with composite aggregation functions
> --
>
> Key: HIVE-20335
> URL: https://issues.apache.org/jira/browse/HIVE-20335
> Project: Hive
>  Issue Type: Test
>  Components: Materialized views, Test
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20326) Create constraints with RELY as default instead of NO RELY

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20326:
---
   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

> Create constraints with RELY as default instead of NO RELY
> --
>
> Key: HIVE-20326
> URL: https://issues.apache.org/jira/browse/HIVE-20326
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20326.1.patch, HIVE-20326.2.patch
>
>
> Currently constraints such as NOT NULL, CHECK are created with ENABLE and NO 
> RELY as default, instead it should be created with ENABLE and RELY as default 
> so that optimizer could take advantage of these constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20333) CBO: Join removal based on PK-FK declared constraints

2018-08-07 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572210#comment-16572210
 ] 

Vineet Garg commented on HIVE-20333:


bq. The rewrite is not applied today and the current PK-FK relationship does 
not allow for a nullable relationship (i.e a declared Foreign Key can't be 
NULL).
So the rewrite for HIVE would be {code:sql} select ss_customer_sk as 
c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales group by 
ss_customer_sk; {code}?

> CBO: Join removal based on PK-FK declared constraints
> -
>
> Key: HIVE-20333
> URL: https://issues.apache.org/jira/browse/HIVE-20333
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
>
> A query of the following shape can have its customer join removed entirely on 
> the basis of the key containment between customer & store_sales.
> {code}
> select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales 
> ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk;
> {code}
> This query after join removal can be encoded in as 
> {code}
> select ss_customer_sk as c_customer_sk,sum(ss_quantity*ss_sales_price) ssales 
> from store_sales where ss_customer_sk is not null group by ss_customer_sk;
> {code}
> The rewrite is not applied today and the current PK-FK relationship does not 
> allow for a nullable relationship (i.e a declared Foreign Key can't be NULL).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20326) Create constraints with RELY as default instead of NO RELY

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20326:
---
Status: Patch Available  (was: Open)

Thanks for looking at it [~jcamachorodriguez]. I have updated the patch and 
have pushed it to master

> Create constraints with RELY as default instead of NO RELY
> --
>
> Key: HIVE-20326
> URL: https://issues.apache.org/jira/browse/HIVE-20326
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20326.1.patch, HIVE-20326.2.patch
>
>
> Currently constraints such as NOT NULL, CHECK are created with ENABLE and NO 
> RELY as default, instead it should be created with ENABLE and RELY as default 
> so that optimizer could take advantage of these constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20326) Create constraints with RELY as default instead of NO RELY

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20326:
---
Attachment: HIVE-20326.2.patch

> Create constraints with RELY as default instead of NO RELY
> --
>
> Key: HIVE-20326
> URL: https://issues.apache.org/jira/browse/HIVE-20326
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20326.1.patch, HIVE-20326.2.patch
>
>
> Currently constraints such as NOT NULL, CHECK are created with ENABLE and NO 
> RELY as default, instead it should be created with ENABLE and RELY as default 
> so that optimizer could take advantage of these constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20326) Create constraints with RELY as default instead of NO RELY

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20326:
---
Status: Open  (was: Patch Available)

> Create constraints with RELY as default instead of NO RELY
> --
>
> Key: HIVE-20326
> URL: https://issues.apache.org/jira/browse/HIVE-20326
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20326.1.patch, HIVE-20326.2.patch
>
>
> Currently constraints such as NOT NULL, CHECK are created with ENABLE and NO 
> RELY as default, instead it should be created with ENABLE and RELY as default 
> so that optimizer could take advantage of these constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572195#comment-16572195
 ] 

Vineet Garg commented on HIVE-20320:


Pushed to master

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572191#comment-16572191
 ] 

Hive QA commented on HIVE-20320:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934684/HIVE-20320.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14867 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13085/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13085/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13085/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934684 - PreCommit-HIVE-Build

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-20327) Compactor should gracefully handle 0 length files and invalid orc files

2018-08-07 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572131#comment-16572131
 ] 

Eugene Koifman edited comment on HIVE-20327 at 8/7/18 7:01 PM:
---

patch 2 is a prototype and an unsuccessful attempt to repro this

Reader deltaReader = OrcFile.createReader(deltaFile, 
OrcFile.readerOptions(conf).maxLength(length));
recordReader = reader.rowsOptions(options, conf);

recordReader.hasNext() returns false when deltaFile is an empty file


was (Author: ekoifman):
patch 2 is a prototype and an unsuccessful attempt to repro this

> Compactor should gracefully handle 0 length files and invalid orc files
> ---
>
> Key: HIVE-20327
> URL: https://issues.apache.org/jira/browse/HIVE-20327
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20327.02.patch
>
>
> Older versions of Streaming API did not handle interrupts well and could 
> leave 0-length ORC files behind which cannot be read.
> These should be just skipped.
> Other cases of file where ORC Reader cannot be created
> 1. regular write (1 txn delta) where the client died and didn't properly 
> close the file - this delta should be aborted and never read
> 2. streaming ingest write (delta_x_y, x < y).  There should always be a side 
> file if the file was not closed properly. (though it may still indicate that 
> length is 0)
> If we check these cases and still can't create a reader, it should not 
> silently skip the file since the system thinks it contains at least some 
> committed data but the file is corrupted (and the side file doesn't point at 
> a valid footer) - we should never be in this situation and we should throw so 
> that the end user can try manual intervention (where the only option may be 
> deleting the file)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572154#comment-16572154
 ] 

Hive QA commented on HIVE-20320:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 64 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
45s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13085/dev-support/hive-personality.sh
 |
| git revision | master / 5391651 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13085/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Nishant Bangarwa (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-20279:

Status: Patch Available  (was: Open)

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Nishant Bangarwa (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-20279:

Attachment: HIVE-20279.1.patch

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.1.patch, HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-19974) Show tables statement includes views and materialized views

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-19974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez resolved HIVE-19974.

Resolution: Fixed

> Show tables statement includes views and materialized views
> ---
>
> Key: HIVE-19974
> URL: https://issues.apache.org/jira/browse/HIVE-19974
> Project: Hive
>  Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
>
> Probably it would be more logical to show only the tables, since there exist 
> 'show views' and 'show materialized views' statements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19408) Improve show materialized views statement to show more information about invalidation

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19408:
---
   Resolution: Fixed
Fix Version/s: 3.2.0
   4.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master, branch-3. Thanks for reviewing [~ashutoshc]

> Improve show materialized views statement to show more information about 
> invalidation
> -
>
> Key: HIVE-19408
> URL: https://issues.apache.org/jira/browse/HIVE-19408
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19408.01.patch, HIVE-19408.02.patch, 
> HIVE-19408.03.patch, HIVE-19408.patch
>
>
> We should show more useful information in addition to materialized view name. 
> For instance, information about whether the materialized view contents are 
> up-to-date or not, and which table(s) have changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-18843) Improve information shown by SHOW MATERIALIZED VIEWS statement

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez resolved HIVE-18843.

Resolution: Fixed

> Improve information shown by SHOW MATERIALIZED VIEWS statement
> --
>
> Key: HIVE-18843
> URL: https://issues.apache.org/jira/browse/HIVE-18843
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently only the materialized view name is shown. We should show some other 
> information such as whether rewriting is enabled for it or if it is 
> up-to-date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20327) Compactor should gracefully handle 0 length files and invalid orc files

2018-08-07 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572131#comment-16572131
 ] 

Eugene Koifman commented on HIVE-20327:
---

patch 2 is a prototype and an unsuccessful attempt to repro this

> Compactor should gracefully handle 0 length files and invalid orc files
> ---
>
> Key: HIVE-20327
> URL: https://issues.apache.org/jira/browse/HIVE-20327
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20327.02.patch
>
>
> Older versions of Streaming API did not handle interrupts well and could 
> leave 0-length ORC files behind which cannot be read.
> These should be just skipped.
> Other cases of file where ORC Reader cannot be created
> 1. regular write (1 txn delta) where the client died and didn't properly 
> close the file - this delta should be aborted and never read
> 2. streaming ingest write (delta_x_y, x < y).  There should always be a side 
> file if the file was not closed properly. (though it may still indicate that 
> length is 0)
> If we check these cases and still can't create a reader, it should not 
> silently skip the file since the system thinks it contains at least some 
> committed data but the file is corrupted (and the side file doesn't point at 
> a valid footer) - we should never be in this situation and we should throw so 
> that the end user can try manual intervention (where the only option may be 
> deleting the file)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Status: Patch Available  (was: In Progress)

> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20332.patch
>
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer 
> (this should be fixed by HIVE-20313). Even if we did, we always assume 
> uniform distribution of the column values, which can easily lead to 
> overestimations on the number of rows read when we filter on 
> {{ROW\_\_ID.writeId}} for materialized views (think about a large transaction 
> for MV creation and then small ones for incremental maintenance). This 
> overestimation can lead to incremental view maintenance not being triggered 
> as cost of the incremental plan is overestimated (we think we will read more 
> rows than we actually do). This could be fixed by introducing histograms that 
> reflect better the column values distribution.
> Till both fixes are implemented, we will use a config variable that will set 
> the selectivity for filter condition on {{ROW\_\_ID}} during the cost 
> calculation. Setting that variable to a low value will favour incremental 
> rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20327) Compactor should gracefully handle 0 length files and invalid orc files

2018-08-07 Thread Eugene Koifman (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-20327:
--
Attachment: HIVE-20327.02.patch

> Compactor should gracefully handle 0 length files and invalid orc files
> ---
>
> Key: HIVE-20327
> URL: https://issues.apache.org/jira/browse/HIVE-20327
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20327.02.patch
>
>
> Older versions of Streaming API did not handle interrupts well and could 
> leave 0-length ORC files behind which cannot be read.
> These should be just skipped.
> Other cases of file where ORC Reader cannot be created
> 1. regular write (1 txn delta) where the client died and didn't properly 
> close the file - this delta should be aborted and never read
> 2. streaming ingest write (delta_x_y, x < y).  There should always be a side 
> file if the file was not closed properly. (though it may still indicate that 
> length is 0)
> If we check these cases and still can't create a reader, it should not 
> silently skip the file since the system thinks it contains at least some 
> committed data but the file is corrupted (and the side file doesn't point at 
> a valid footer) - we should never be in this situation and we should throw so 
> that the end user can try manual intervention (where the only option may be 
> deleting the file)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Attachment: HIVE-20332.patch

> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20332.patch
>
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer 
> (this should be fixed by HIVE-20313). Even if we did, we always assume 
> uniform distribution of the column values, which can easily lead to 
> overestimations on the number of rows read when we filter on 
> {{ROW\_\_ID.writeId}} for materialized views (think about a large transaction 
> for MV creation and then small ones for incremental maintenance). This 
> overestimation can lead to incremental view maintenance not being triggered 
> as cost of the incremental plan is overestimated (we think we will read more 
> rows than we actually do). This could be fixed by introducing histograms that 
> reflect better the column values distribution.
> Till both fixes are implemented, we will use a config variable that will set 
> the selectivity for filter condition on {{ROW\_\_ID}} during the cost 
> calculation. Setting that variable to a low value will favour incremental 
> rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19408) Improve show materialized views statement to show more information about invalidation

2018-08-07 Thread Ashutosh Chauhan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572128#comment-16572128
 ] 

Ashutosh Chauhan commented on HIVE-19408:
-

+1

> Improve show materialized views statement to show more information about 
> invalidation
> -
>
> Key: HIVE-19408
> URL: https://issues.apache.org/jira/browse/HIVE-19408
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-19408.01.patch, HIVE-19408.02.patch, 
> HIVE-19408.03.patch, HIVE-19408.patch
>
>
> We should show more useful information in addition to materialized view name. 
> For instance, information about whether the materialized view contents are 
> up-to-date or not, and which table(s) have changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20279) HiveContextAwareRecordReader slows down Druid Scan queries.

2018-08-07 Thread Nishant Bangarwa (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishant Bangarwa updated HIVE-20279:

Attachment: HIVE-20279.patch

> HiveContextAwareRecordReader slows down Druid Scan queries. 
> 
>
> Key: HIVE-20279
> URL: https://issues.apache.org/jira/browse/HIVE-20279
> Project: Hive
>  Issue Type: Improvement
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20279.patch, scan2.svg
>
>
> HiveContextAwareRecordReader add lots of overhead for Druid Scan Queries. 
> See attached flame graph. 
> Looks like the operations for checking for existence of footer/header buffer 
> takes most of time For druid and other storage handlers that do not have 
> footer buffer we should skip the logic for checking the existence for storage 
> handlers atleast. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19408) Improve show materialized views statement to show more information about invalidation

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572096#comment-16572096
 ] 

Hive QA commented on HIVE-19408:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934677/HIVE-19408.03.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14868 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13084/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13084/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13084/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934677 - PreCommit-HIVE-Build

> Improve show materialized views statement to show more information about 
> invalidation
> -
>
> Key: HIVE-19408
> URL: https://issues.apache.org/jira/browse/HIVE-19408
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-19408.01.patch, HIVE-19408.02.patch, 
> HIVE-19408.03.patch, HIVE-19408.patch
>
>
> We should show more useful information in addition to materialized view name. 
> For instance, information about whether the materialized view contents are 
> up-to-date or not, and which table(s) have changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Description: 
Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer 
(this should be fixed by HIVE-20313). Even if we did, we always assume uniform 
distribution of the column values, which can easily lead to overestimations on 
the number of rows read when we filter on {{ROW\_\_ID.writeId}} for 
materialized views (think about a large transaction for MV creation and then 
small ones for incremental maintenance). This overestimation can lead to 
incremental view maintenance not being triggered as cost of the incremental 
plan is overestimated (we think we will read more rows than we actually do). 
This could be fixed by introducing histograms that reflect better the column 
values distribution.

Till both fixes are implemented, we will use a config variable that will set 
the selectivity for filter condition on {{ROW\_\_ID}} during the cost 
calculation. Setting that variable to a low value will favour incremental 
rebuild over full rebuild.

  was:
Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW\_\_ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.


> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer 
> (this should be fixed by HIVE-20313). Even if we did, we always assume 
> uniform distribution of the column values, which can easily lead to 
> overestimations on the number of rows read when we filter on 
> {{ROW\_\_ID.writeId}} for materialized views (think about a large transaction 
> for MV creation and then small ones for incremental maintenance). This 
> overestimation can lead to incremental view maintenance not being triggered 
> as cost of the incremental plan is overestimated (we think we will read more 
> rows than we actually do). This could be fixed by introducing histograms that 
> reflect better the column values distribution.
> Till both fixes are implemented, we will use a config variable that will set 
> the selectivity for filter condition on {{ROW\_\_ID}} during the cost 
> calculation. Setting that variable to a low value will favour incremental 
> rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572088#comment-16572088
 ] 

Jesus Camacho Rodriguez commented on HIVE-20332:


[~ekoifman], agree. HIVE-20313 plus actual column values distribution 
information will be needed in the longer term to make this a cost-based 
decision instead of a heuristic one.

> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the 
> optimizer. Even if we did, we always assume uniform distribution of the 
> column values, which can easily lead to overestimations on the number of rows 
> read when we filter on {{ROW\_\_ID.writeId}} for materialized views (think 
> about a large transaction for MV creation and then small ones for incremental 
> maintenance). This overestimation can lead to incremental view maintenance 
> not being triggered as cost of the incremental plan is overestimated (we 
> think we will read more rows than we actually do). This could be fixed by 
> introducing histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting 
> that variable to a low value will favour incremental rebuild over full 
> rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572085#comment-16572085
 ] 

Eugene Koifman commented on HIVE-20332:
---

HIVE-20313 should be considered (though hard to say how much effort this would 
be)

> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the 
> optimizer. Even if we did, we always assume uniform distribution of the 
> column values, which can easily lead to overestimations on the number of rows 
> read when we filter on {{ROW\_\_ID.writeId}} for materialized views (think 
> about a large transaction for MV creation and then small ones for incremental 
> maintenance). This overestimation can lead to incremental view maintenance 
> not being triggered as cost of the incremental plan is overestimated (we 
> think we will read more rows than we actually do). This could be fixed by 
> introducing histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting 
> that variable to a low value will favour incremental rebuild over full 
> rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Description: 
Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW__ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.

  was:
Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW\_\_ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on ROW__ID during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.


> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
> Even if we did, we always assume uniform distribution of the column values, 
> which can easily lead to overestimations on the number of rows read when we 
> filter on {{ROW__ID.writeId}} for materialized views (think about a large 
> transaction for MV creation and then small ones for incremental maintenance). 
> This overestimation can lead to incremental view maintenance not being 
> triggered as cost of the incremental plan is overestimated (we think we will 
> read more rows than we actually do). This could be fixed by introducing 
> histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting 
> that variable to a low value will favour incremental rebuild over full 
> rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Description: 
Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW\_\_ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.

  was:
Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW__ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.


> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the 
> optimizer. Even if we did, we always assume uniform distribution of the 
> column values, which can easily lead to overestimations on the number of rows 
> read when we filter on {{ROW\_\_ID.writeId}} for materialized views (think 
> about a large transaction for MV creation and then small ones for incremental 
> maintenance). This overestimation can lead to incremental view maintenance 
> not being triggered as cost of the incremental plan is overestimated (we 
> think we will read more rows than we actually do). This could be fixed by 
> introducing histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on {{ROW\_\_ID}} during the cost calculation. Setting 
> that variable to a low value will favour incremental rebuild over full 
> rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-20332:
---
Description: 
Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW\_\_ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on ROW__ID during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.

  was:
Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
Even if we did, we always assume uniform distribution of the column values, 
which can easily lead to overestimations on the number of rows read when we 
filter on {{ROW__ID.writeId}} for materialized views (think about a large 
transaction for MV creation and then small ones for incremental maintenance). 
This overestimation can lead to incremental view maintenance not being 
triggered as cost of the incremental plan is overestimated (we think we will 
read more rows than we actually do). This could be fixed by introducing 
histograms that reflect better the column values distribution.

Till that moment, we will use a config variable that will set the selectivity 
for filter condition on ROW__ID during the cost calculation. Setting that 
variable to a low value will favour incremental rebuild over full rebuild.


> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW\_\_ID.writeId}} to the 
> optimizer. Even if we did, we always assume uniform distribution of the 
> column values, which can easily lead to overestimations on the number of rows 
> read when we filter on {{ROW\_\_ID.writeId}} for materialized views (think 
> about a large transaction for MV creation and then small ones for incremental 
> maintenance). This overestimation can lead to incremental view maintenance 
> not being triggered as cost of the incremental plan is overestimated (we 
> think we will read more rows than we actually do). This could be fixed by 
> introducing histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on ROW__ID during the cost calculation. Setting that 
> variable to a low value will favour incremental rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-20332 started by Jesus Camacho Rodriguez.
--
> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
> Even if we did, we always assume uniform distribution of the column values, 
> which can easily lead to overestimations on the number of rows read when we 
> filter on {{ROW__ID.writeId}} for materialized views (think about a large 
> transaction for MV creation and then small ones for incremental maintenance). 
> This overestimation can lead to incremental view maintenance not being 
> triggered as cost of the incremental plan is overestimated (we think we will 
> read more rows than we actually do). This could be fixed by introducing 
> histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on ROW__ID during the cost calculation. Setting that 
> variable to a low value will favour incremental rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20332) Materialized views: Introduce heuristic on selectivity over ROW__ID to favour incremental rebuild

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-20332:
--


> Materialized views: Introduce heuristic on selectivity over ROW__ID to favour 
> incremental rebuild
> -
>
> Key: HIVE-20332
> URL: https://issues.apache.org/jira/browse/HIVE-20332
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Currently, we do not expose stats over {{ROW__ID.writeId}} to the optimizer. 
> Even if we did, we always assume uniform distribution of the column values, 
> which can easily lead to overestimations on the number of rows read when we 
> filter on {{ROW__ID.writeId}} for materialized views (think about a large 
> transaction for MV creation and then small ones for incremental maintenance). 
> This overestimation can lead to incremental view maintenance not being 
> triggered as cost of the incremental plan is overestimated (we think we will 
> read more rows than we actually do). This could be fixed by introducing 
> histograms that reflect better the column values distribution.
> Till that moment, we will use a config variable that will set the selectivity 
> for filter condition on ROW__ID during the cost calculation. Setting that 
> variable to a low value will favour incremental rebuild over full rebuild.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19408) Improve show materialized views statement to show more information about invalidation

2018-08-07 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572040#comment-16572040
 ] 

Hive QA commented on HIVE-19408:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
47s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 12 new + 530 unchanged - 6 
fixed = 542 total (was 536) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
53s{color} | {color:red} ql generated 1 new + 2304 unchanged - 0 fixed = 2305 
total (was 2304) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  org.apache.hadoop.hive.ql.metadata.Hive.isOutdatedMaterializedView(Table, 
ValidTxnWriteIdList, long, List, boolean) has Boolean return type and returns 
explicit null  At Hive.java:has Boolean return type and returns explicit null  
At Hive.java:[line 1553] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13084/dev-support/hive-personality.sh
 |
| git revision | master / 5391651 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13084/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13084/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13084/yetus/new-findbugs-ql.html
 |
| modules | C: itests itests/hive-unit ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13084/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Improve show materialized views statement to show more information about 
> invalidation
> -
>
> Key: HIVE-19408
> URL: https://issues.apache.org/jira/browse/HIVE-19408
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 3.0.0
>

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
Status: Patch Available  (was: Open)

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
Attachment: HIVE-20320.3.patch

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
Attachment: (was: HIVE-20320.3.patch)

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
Status: Open  (was: Patch Available)

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20320) Turn on hive.optimize.remove.sq_count_check flag

2018-08-07 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20320:
---
Attachment: HIVE-20320.3.patch

> Turn on hive.optimize.remove.sq_count_check flag
> 
>
> Key: HIVE-20320
> URL: https://issues.apache.org/jira/browse/HIVE-20320
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20320.1.patch, HIVE-20320.2.patch, 
> HIVE-20320.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20331) Query with union all, lateral view and Join fails with "cannot find parent in the child operator"

2018-08-07 Thread Aihua Xu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-20331:
---


> Query with union all, lateral view and Join fails with "cannot find parent in 
> the child operator"
> -
>
> Key: HIVE-20331
> URL: https://issues.apache.org/jira/browse/HIVE-20331
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>
> The following query with Union, Lateral view and Join will fail during 
> execution with the exception below.
> {noformat}
> create table t1(col1 int);
> SELECT 1 AS `col1`
> FROM t1
> UNION ALL
>   SELECT 2 AS `col1`
>   FROM
> (SELECT col1
>  FROM t1
> ) x1
> JOIN
>   (SELECT col1
>   FROM
> (SELECT 
>   Row_Number() over (PARTITION BY col1 ORDER BY col1) AS `col1`
> FROM t1
> ) x2 lateral VIEW explode(map(10,1))`mapObj` AS `col2`, `col3`
>   ) `expdObj`  
> {noformat}
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive internal 
> error: cannot find parent in the child operator!
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:509)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {noformat}
> After debugging, seems we have issues in GenMRFileSink1 class in which we are 
> setting incorrect aliasToWork to the MapWork.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20326) Create constraints with RELY as default instead of NO RELY

2018-08-07 Thread Jesus Camacho Rodriguez (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572012#comment-16572012
 ] 

Jesus Camacho Rodriguez commented on HIVE-20326:


+1

Can you update the comment in line 1001 while committing? Thanks

> Create constraints with RELY as default instead of NO RELY
> --
>
> Key: HIVE-20326
> URL: https://issues.apache.org/jira/browse/HIVE-20326
> Project: Hive
>  Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20326.1.patch
>
>
> Currently constraints such as NOT NULL, CHECK are created with ENABLE and NO 
> RELY as default, instead it should be created with ENABLE and RELY as default 
> so that optimizer could take advantage of these constraints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

1 2 >

1 - 100 of 149 matches

Mail list logo