[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877505#comment-16877505
]
Hive QA commented on HIVE-21921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973484/HIVE-21921.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16360 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17826/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17826/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17826/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973484 - PreCommit-HIVE-Build
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21944: --- Attachment: HIVE-21944.1.patch > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877486#comment-16877486
]
Hive QA commented on HIVE-21921:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 5 new + 153 unchanged - 1
fixed = 158 total (was 154) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17826/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17826/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17826/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877475#comment-16877475
]
Hive QA commented on HIVE-18842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973481/HIVE-18842.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 16363 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17825/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17825/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17825/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973481 - PreCommit-HIVE-Build
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Status: Open (was: Patch Available) > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Attachment: HIVE-21838.5.patch > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Status: Patch Available (was: Open) > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877456#comment-16877456
]
Hive QA commented on HIVE-18842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 9 new + 708 unchanged - 1
fixed = 717 total (was 709) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
3s{color} | {color:red} ql generated 2 new + 2253 unchanged - 1 fixed = 2255
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to LA29_5 in
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At
HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At HiveLexer.java:[line 12954] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA242 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48940-48953] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17825/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17825/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17825/yetus/new-findbugs-ql.html
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17825/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877442#comment-16877442
]
Hive QA commented on HIVE-21909:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973453/HIVE-21909.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16362 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=350)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17824/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17824/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17824/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973453 - PreCommit-HIVE-Build
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877428#comment-16877428
]
Hive QA commented on HIVE-21923:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 22m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} common: The patch generated 1 new + 1914 unchanged - 0
fixed = 1915 total (was 1914) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 20m
11s{color} | {color:red} root: The patch generated 1 new + 73200 unchanged - 0
fixed = 73201 total (was 73200) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m
2s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
35s{color} | {color:red} ql in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17823/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus/diff-checkstyle-root.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus/patch-findbugs-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus/patch-findbugs-ql.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus/patch-javadoc-ql.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17823/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
>
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877427#comment-16877427
]
Hive QA commented on HIVE-21909:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
21s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
37s{color} | {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color}
| {color:red} service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} llap-client: The patch generated 10 new + 30 unchanged
- 1 fixed = 40 total (was 31) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} llap-server: The patch generated 15 new + 50 unchanged
- 0 fixed = 65 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
18s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17824/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17824/yetus/patch-mvninstall-llap-server.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17824/yetus/patch-mvninstall-service.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17824/yetus/patch-compile-llap-server.txt
|
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877407#comment-16877407
]
Hive QA commented on HIVE-21923:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973452/HIVE-21923.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16359 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17823/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17823/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17823/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973452 - PreCommit-HIVE-Build
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch,
> HIVE-21923.03.patch, HIVE-21923.04.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21943) Why do we not have audit log in HiveServer2?
[ https://issues.apache.org/jira/browse/HIVE-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HIVE-21943: -- Description: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would assoicate with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? We could add much infomations in the audit log of HS2, like username, ipAddress, queryId, queryStr, etc. was: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would assocate with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? We could add much infomations in the audit log of HS2, like username, ipAddress, queryId, queryStr, etc. > Why do we not have audit log in HiveServer2? > > > Key: HIVE-21943 > URL: https://issues.apache.org/jira/browse/HIVE-21943 > Project: Hive > Issue Type: Improvement >Reporter: Hui An >Priority: Major > > Currently, we have audit log in the metastore( see > [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides > us useful info, like metadata operations, ip address, etc. But when we use > HS2 to connect HMS, all operations would assoicate with HS2 ip address, not > actual user's ip address. This could cause difficult problems when we want to > locate user's ip address. So I am thinking if we could add audit log in the > HS2 too? > We could add much infomations in the audit log of HS2, like username, > ipAddress, queryId, queryStr, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271306
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299735367
##
File path:
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemon.java
##
@@ -0,0 +1,111 @@
+package org.apache.hadoop.hive.llap.daemon.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.LlapDaemonInfo;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.metrics.LlapMetricsSystem;
+import org.apache.hadoop.hive.llap.metrics.MetricsUtils;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Captor;
+import org.mockito.Mock;
+import org.mockito.Spy;
+import org.mockito.internal.util.reflection.Fields;
+import org.mockito.internal.util.reflection.InstanceField;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.mockito.MockitoAnnotations.initMocks;
+
+public class TestLlapDaemon {
+
+ private static final String[] METRICS_SOURCES = new String[]{
+"JvmMetrics",
+"LlapDaemonExecutorMetrics-" + MetricsUtils.getHostName(),
+"LlapDaemonJvmMetrics-" + MetricsUtils.getHostName(),
+MetricsUtils.METRICS_PROCESS_NAME
+ };
+
+ @Spy
Review comment:
Why is this a Spy or more important even a mock. You could just use the
standard HiveConf instance and store the two config values for zookeeper
there...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271306)
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271308 ] ASF GitHub Bot logged work on HIVE-21909: - Author: ASF GitHub Bot Created on: 03/Jul/19 00:36 Start Date: 03/Jul/19 00:36 Worklog Time Spent: 10m Work Description: odraese commented on pull request #695: HIVE-21909: Publish the LLAP Daemon capacity through ZooKeeper and ho… URL: https://github.com/apache/hive/pull/695#discussion_r299605545 ## File path: llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java ## @@ -70,6 +70,7 @@ private SlotZnode slotZnode; + private ServiceRecord srv; Review comment: Maybe rename _srv_ to something more descriptive This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271308) Time Spent: 40m (was: 0.5h) > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271309 ] ASF GitHub Bot logged work on HIVE-21909: - Author: ASF GitHub Bot Created on: 03/Jul/19 00:36 Start Date: 03/Jul/19 00:36 Worklog Time Spent: 10m Work Description: odraese commented on pull request #695: HIVE-21909: Publish the LLAP Daemon capacity through ZooKeeper and ho… URL: https://github.com/apache/hive/pull/695#discussion_r299733175 ## File path: llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemon.java ## @@ -0,0 +1,111 @@ +package org.apache.hadoop.hive.llap.daemon.impl; Review comment: Apache license missing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271309) Time Spent: 40m (was: 0.5h) > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271305
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299732376
##
File path:
llap-client/src/test/org/apache/hadoop/hive/llap/registry/impl/TestLlapZookeeperRegistryImpl.java
##
@@ -0,0 +1,124 @@
+package org.apache.hadoop.hive.llap.registry.impl;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.RetryOneTime;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.registry.ServiceInstanceSet;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.internal.util.reflection.Fields;
+import org.mockito.internal.util.reflection.InstanceField;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static java.lang.Integer.parseInt;
+import static org.junit.Assert.assertEquals;
+
+public class TestLlapZookeeperRegistryImpl {
+
+ @Spy
+ private HiveConf mockConf = new HiveConf();
+
+ private LlapZookeeperRegistryImpl registry;
+
+ private CuratorFramework curatorFramework;
+ private TestingServer server;
+
+ @Before
+ public void setUp() throws Exception {
+MockitoAnnotations.initMocks(this);
+registry = new LlapZookeeperRegistryImpl("TestLlapZookeeperRegistryImpl",
mockConf);
+
+server = new TestingServer();
+server.start();
+
+curatorFramework = CuratorFrameworkFactory.
+builder().
+connectString(server.getConnectString()).
+sessionTimeoutMs(1000).
+retryPolicy(new RetryOneTime(1000)).
+build();
+curatorFramework.start();
+
+trySetMock(registry, CuratorFramework.class, curatorFramework);
+
+ }
+
+ @After
+ public void tearDown() throws IOException {
+curatorFramework.close();
+server.stop();
+ }
+
+ @Test
+ public void testRegister() throws Exception {
+// Given
+int expectedExecutorCount = HiveConf.getIntVar(mockConf,
HiveConf.ConfVars.LLAP_DAEMON_NUM_EXECUTORS);
+int expectedQueueSize = HiveConf.getIntVar(mockConf,
HiveConf.ConfVars.LLAP_DAEMON_TASK_SCHEDULER_WAIT_QUEUE_SIZE);
+
+// When
+registry.register();
+ServiceInstanceSet serviceInstanceSet =
+ registry.getInstances("LLAP", 1000);
+
+// Then
+Collection llaps = serviceInstanceSet.getAll();
+assertEquals(1, llaps.size());
+LlapServiceInstance serviceInstance = llaps.iterator().next();
+Map attributes = serviceInstance.getProperties();
+
+assertEquals(expectedQueueSize,
+
parseInt(attributes.get(LlapRegistryService.LLAP_DAEMON_TASK_SCHEDULER_ENABLED_WAIT_QUEUE_SIZE)));
+assertEquals(expectedExecutorCount,
+
parseInt(attributes.get(LlapRegistryService.LLAP_DAEMON_NUM_ENABLED_EXECUTORS)));
+ }
+
+ @Test
+ public void testUpdate() throws Exception {
+// Given
+String expectedExecutorCount = "2";
+String expectedQueueSize = "20";
+Map capacityValues = new HashMap<>(2);
+capacityValues.put(LlapRegistryService.LLAP_DAEMON_NUM_ENABLED_EXECUTORS,
expectedExecutorCount);
+
capacityValues.put(LlapRegistryService.LLAP_DAEMON_TASK_SCHEDULER_ENABLED_WAIT_QUEUE_SIZE,
expectedQueueSize);
+
+// When
+registry.register();
+registry.updateRegistration(capacityValues.entrySet());
+ServiceInstanceSet serviceInstanceSet =
+registry.getInstances("LLAP", 1000);
+
+// Then
+Collection llaps = serviceInstanceSet.getAll();
+assertEquals(1, llaps.size());
+LlapServiceInstance serviceInstance = llaps.iterator().next();
+Map attributes = serviceInstance.getProperties();
+
+assertEquals(expectedQueueSize,
+
attributes.get(LlapRegistryService.LLAP_DAEMON_TASK_SCHEDULER_ENABLED_WAIT_QUEUE_SIZE));
+assertEquals(expectedExecutorCount,
+
attributes.get(LlapRegistryService.LLAP_DAEMON_NUM_ENABLED_EXECUTORS));
+ }
+
+ static void trySetMock(Object o, Class clazz, T mock) {
+List instanceFields = Fields
+ .allDeclaredFieldsOf(o)
+ .filter(instanceField ->
!clazz.isAssignableFrom(instanceField.jdkField().getType()))
+ .instanceFields();
+if (instanceFields.size() != 1) {
Review comment:
Wouldn't
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271310
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299736463
##
File path:
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##
@@ -2500,11 +2504,11 @@ public void shutdown() {
private int numPreemptedTasks = 0;
Review comment:
Is numPreemtedTasks used/read anywhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271310)
Time Spent: 50m (was: 40m)
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271304
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299604870
##
File path:
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
##
@@ -132,6 +136,12 @@ private void unregisterWorker() throws IOException {
}
}
+ public void updateRegistration(Iterable>
attributes) throws IOException {
+if (isDaemon && this.registry != null) {
Review comment:
Question: Why only if isDaemon=true? Other register/unregister only care
about registry != null...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271304)
Time Spent: 0.5h (was: 20m)
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271311 ] ASF GitHub Bot logged work on HIVE-21909: - Author: ASF GitHub Bot Created on: 03/Jul/19 00:36 Start Date: 03/Jul/19 00:36 Worklog Time Spent: 10m Work Description: odraese commented on pull request #695: HIVE-21909: Publish the LLAP Daemon capacity through ZooKeeper and ho… URL: https://github.com/apache/hive/pull/695#discussion_r299733084 ## File path: llap-client/src/test/org/apache/hadoop/hive/llap/registry/impl/TestLlapZookeeperRegistryImpl.java ## @@ -0,0 +1,124 @@ +package org.apache.hadoop.hive.llap.registry.impl; Review comment: Apache license missing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271311) > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271303
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299731966
##
File path:
llap-client/src/test/org/apache/hadoop/hive/llap/registry/impl/TestLlapZookeeperRegistryImpl.java
##
@@ -0,0 +1,124 @@
+package org.apache.hadoop.hive.llap.registry.impl;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.RetryOneTime;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.registry.ServiceInstanceSet;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.MockitoAnnotations;
+import org.mockito.Spy;
+import org.mockito.internal.util.reflection.Fields;
+import org.mockito.internal.util.reflection.InstanceField;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static java.lang.Integer.parseInt;
+import static org.junit.Assert.assertEquals;
+
+public class TestLlapZookeeperRegistryImpl {
+
+ @Spy
Review comment:
Why spy on HiveConf here? Doesn't seem to get used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271303)
Time Spent: 20m (was: 10m)
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[
https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271307
]
ASF GitHub Bot logged work on HIVE-21909:
-
Author: ASF GitHub Bot
Created on: 03/Jul/19 00:36
Start Date: 03/Jul/19 00:36
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #695: HIVE-21909:
Publish the LLAP Daemon capacity through ZooKeeper and ho…
URL: https://github.com/apache/hive/pull/695#discussion_r299603003
##
File path:
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
##
@@ -112,6 +112,11 @@ public void unregister() throws IOException {
// nothing to unregister
}
+ @Override
+ public void updateRegistration(Iterable attributes) throws IOException {
Review comment:
The Iterable should be parameterized. (Iterable>)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271307)
Time Spent: 40m (was: 0.5h)
> Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity
> when scheduling new tasks
> ---
>
> Key: HIVE-21909
> URL: https://issues.apache.org/jira/browse/HIVE-21909
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21909.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The changes in LlapDaemonNode status (or executor number) should be reflected
> in the ZooKeeper information, and we should make sure TezAMs are reading and
> honor the changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[
https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877378#comment-16877378
]
Hive QA commented on HIVE-21838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973445/HIVE-21838.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17822/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17822/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17822/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/ShimLoader.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/CancellationException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/sql/DriverManager.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/sql/SQLException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/jdbc/target/hive-jdbc-4.0.0-SNAPSHOT.jar(org/apache/hive/jdbc/HiveConnection.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/conf/Configured.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/NullWritable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/Text.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobClient.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobConf.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/Job.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/JobID.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.class)]]
[loading
[jira] [Updated] (HIVE-21571) SHOW COMPACTIONS shows column names as its first output row
[ https://issues.apache.org/jira/browse/HIVE-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21571: -- Status: Open (was: Patch Available) > SHOW COMPACTIONS shows column names as its first output row > --- > > Key: HIVE-21571 > URL: https://issues.apache.org/jira/browse/HIVE-21571 > Project: Hive > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21571.01.patch, HIVE-21571.02.patch, > HIVE-21571.03.patch, HIVE-21571.04.patch, HIVE-21571.05.patch, > HIVE-21571.patch > > > SHOW COMPACTIONS yields a resultset with nice column names, and then the > first row of data is a repetition of those column names. This is somewhat > confusing and hard to read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21571) SHOW COMPACTIONS shows column names as its first output row
[ https://issues.apache.org/jira/browse/HIVE-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21571: -- Attachment: HIVE-21571.05.patch Status: Patch Available (was: Open) > SHOW COMPACTIONS shows column names as its first output row > --- > > Key: HIVE-21571 > URL: https://issues.apache.org/jira/browse/HIVE-21571 > Project: Hive > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21571.01.patch, HIVE-21571.02.patch, > HIVE-21571.03.patch, HIVE-21571.04.patch, HIVE-21571.05.patch, > HIVE-21571.patch > > > SHOW COMPACTIONS yields a resultset with nice column names, and then the > first row of data is a repetition of those column names. This is somewhat > confusing and hard to read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[
https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877375#comment-16877375
]
Hive QA commented on HIVE-21944:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973439/HIVE-21944.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16330 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17821/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17821/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17821/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973439 - PreCommit-HIVE-Build
> Remove unused methods, fields and variables from Vectorizer
> ---
>
> Key: HIVE-21944
> URL: https://issues.apache.org/jira/browse/HIVE-21944
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Trivial
> Attachments: HIVE-21944.1.patch
>
>
> It seems there are many unused fields, variables and methods in
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them
> would make the code easier to understand.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?focusedWorklogId=271291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271291 ] ASF GitHub Bot logged work on HIVE-21947: - Author: ASF GitHub Bot Created on: 03/Jul/19 00:03 Start Date: 03/Jul/19 00:03 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #697: HIVE-21947 Move Materialized View Update under DDL URL: https://github.com/apache/hive/pull/697#discussion_r299730674 ## File path: ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out ## @@ -227,7 +227,9 @@ STAGE PLANS: Table: default.cmv_mat_view Stage: Stage-5 -Materialized View Work +Materialized View Update + name: default.cmv_mat_view + retrievAndInclude: true Review comment: Thanks, nice catch, fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271291) Time Spent: 0.5h (was: 20m) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: (was: HIVE-21947.01.patch) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: HIVE-21947.01.patch > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877370#comment-16877370 ] Jesus Camacho Rodriguez commented on HIVE-21947: Thanks [~mgergely]. Left a minor comment. +1 (pending tests) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?focusedWorklogId=271285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271285 ] ASF GitHub Bot logged work on HIVE-21947: - Author: ASF GitHub Bot Created on: 02/Jul/19 23:50 Start Date: 02/Jul/19 23:50 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #697: HIVE-21947 Move Materialized View Update under DDL URL: https://github.com/apache/hive/pull/697#discussion_r299728181 ## File path: ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_3.q.out ## @@ -227,7 +227,9 @@ STAGE PLANS: Table: default.cmv_mat_view Stage: Stage-5 -Materialized View Work +Materialized View Update + name: default.cmv_mat_view + retrievAndInclude: true Review comment: Typo? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271285) Time Spent: 20m (was: 10m) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877366#comment-16877366 ] Miklos Gergely commented on HIVE-21947: --- [~jcamachorodriguez] created [https://github.com/apache/hive/pull/697]. Just realised that accidentally added 3 files which were not supposed to be part of the patch, so I've uploaded here a new patch. Thanks for reviewing. > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?focusedWorklogId=271280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271280 ] ASF GitHub Bot logged work on HIVE-21947: - Author: ASF GitHub Bot Created on: 02/Jul/19 23:40 Start Date: 02/Jul/19 23:40 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #697: HIVE-21947 Move Materialized View Update under DDL URL: https://github.com/apache/hive/pull/697 MaterializedViewDesc is not part of the DDL framework, it has it's own task, which defines itself as DDL. They should be moved to the framework. Also cleaned up a bit the alter materialized view handling - no need for types now, there is only one type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271280) Time Spent: 10m Remaining Estimate: 0h > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21947: -- Labels: pull-request-available refactor-ddl (was: refactor-ddl) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: HIVE-21947.01.patch > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: (was: HIVE-21947.01.patch) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: HIVE-21947.01.patch > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: (was: HIVE-21947.01.patch) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877361#comment-16877361 ] Jesus Camacho Rodriguez commented on HIVE-21947: [~mgergely], can you create a PR for this issue so I can review? Thanks > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Attachment: HIVE-21947.01.patch > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21947: -- Status: Patch Available (was: Open) > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21947.01.patch > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21947) Move Materialized View Update under DDL
[ https://issues.apache.org/jira/browse/HIVE-21947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21947: - > Move Materialized View Update under DDL > --- > > Key: HIVE-21947 > URL: https://issues.apache.org/jira/browse/HIVE-21947 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > > MaterializedViewDesc is not part of the DDL framework, it has it's own task, > which defines itself as DDL. They should be moved to the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[
https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877355#comment-16877355
]
Hive QA commented on HIVE-21944:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 399 unchanged - 2
fixed = 399 total (was 401) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} ql generated 0 new + 2248 unchanged - 6 fixed = 2248
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17821/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17821/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove unused methods, fields and variables from Vectorizer
> ---
>
> Key: HIVE-21944
> URL: https://issues.apache.org/jira/browse/HIVE-21944
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Trivial
> Attachments: HIVE-21944.1.patch
>
>
> It seems there are many unused fields, variables and methods in
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them
> would make the code easier to understand.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21946) Consider data organization of a materialized view in transparent rewriting
[ https://issues.apache.org/jira/browse/HIVE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21946: --- Summary: Consider data organization of a materialized view in transparent rewriting (was: Consider data distribution of a materialized view in transparent rewriting) > Consider data organization of a materialized view in transparent rewriting > -- > > Key: HIVE-21946 > URL: https://issues.apache.org/jira/browse/HIVE-21946 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, we do consider partitioning of the original table, but we do not > take into account data organization (DISTRIBUTE/SORT/CLUSTER) in the > optimizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21946) Consider data organization of a materialized view in transparent rewriting
[ https://issues.apache.org/jira/browse/HIVE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21946: --- Description: Currently, we do consider partitioning of the original table, but we do not take into account data organization (DISTRIBUTE/SORT/CLUSTER) during the optimization phase. (was: Currently, we do consider partitioning of the original table, but we do not take into account data organization (DISTRIBUTE/SORT/CLUSTER) in the optimizer.) > Consider data organization of a materialized view in transparent rewriting > -- > > Key: HIVE-21946 > URL: https://issues.apache.org/jira/browse/HIVE-21946 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, we do consider partitioning of the original table, but we do not > take into account data organization (DISTRIBUTE/SORT/CLUSTER) during the > optimization phase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21946) Consider data distribution of a materialized view in transparent rewriting
[ https://issues.apache.org/jira/browse/HIVE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21946: -- > Consider data distribution of a materialized view in transparent rewriting > -- > > Key: HIVE-21946 > URL: https://issues.apache.org/jira/browse/HIVE-21946 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, we do consider partitioning of the original table, but we do not > take into account data organization (DISTRIBUTE/SORT/CLUSTER) in the > optimizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877343#comment-16877343
]
Hive QA commented on HIVE-21880:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973406/HIVE-21880.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16362 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17820/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17820/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17820/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973406 - PreCommit-HIVE-Build
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21880.01.patch, HIVE-21880.02.patch,
> HIVE-21880.03.patch, HIVE-21880.04.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Need tp enable
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
> which is disabled as it is flaky and randomly failing with below error.
> {code}
> Error Message
> Notification events are missing in the meta store.
> Stacktrace
> java.lang.IllegalStateException: Notification events are missing in the meta
> store.
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
> at
> org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(ReplDumpTask.java:231)
> at
> org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTask.java:121)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInstance.java:227)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:282)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:265)
> at
> org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseInstance.java:289)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestReplicationScenariosAcidTablesBootstrap.java:328)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Commented] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877342#comment-16877342 ] Jesus Camacho Rodriguez commented on HIVE-21921: +1 (pending tests) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
[
https://issues.apache.org/jira/browse/HIVE-21880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877330#comment-16877330
]
Hive QA commented on HIVE-21880:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
29s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17820/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17820/yetus/diff-checkstyle-itests_hcatalog-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql hcatalog/server-extensions
itests/hcatalog-unit itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17820/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable flaky test
> TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites.
> ---
>
> Key: HIVE-21880
> URL: https://issues.apache.org/jira/browse/HIVE-21880
>
[jira] [Commented] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[
https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877307#comment-16877307
]
Hive QA commented on HIVE-21910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973399/HIVE-21910.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16311 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingNonLeader.testHouseKeepingThreadExistence
(batchId=242)
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17819/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17819/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17819/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973399 - PreCommit-HIVE-Build
> Multiple target location generation in HostAffinitySplitLocationProvider
>
>
> Key: HIVE-21910
> URL: https://issues.apache.org/jira/browse/HIVE-21910
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21910.2.patch, HIVE-21910.3.patch,
> HIVE-21910.4.patch, HIVE-21910.5.patch, HIVE-21910.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We need to generate multiple target locations by
> HostAffinitySplitLocationProvider, so we will have deterministic fallback
> nodes in case the target node is disabled
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Patch Available (was: Open) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Status: Open (was: Patch Available) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271227
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 21:09
Start Date: 02/Jul/19 21:09
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #693:
HIVE-21921: Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299683240
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -230,33 +228,59 @@ private RexNode rewriteScalar(RelMetadataQuery mq,
RexSubQuery e, Set
variablesSet,
HiveSubQRemoveRelBuilder builder) {
final SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
-assert(op == SqlStdOperatorTable.SOME_GE
-|| op == SqlStdOperatorTable.SOME_LE
-|| op == SqlStdOperatorTable.SOME_LT
-|| op == SqlStdOperatorTable.SOME_GT);
-builder.push(e.rel)
-.aggregate(builder.groupKey(),
-op.comparisonKind == SqlKind.GREATER_THAN
-|| op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
-? builder.min("m", builder.field(0))
-: builder.max("m", builder.field(0)),
-builder.count(false, "c"),
-builder.count(false, "d", builder.field(0)))
-.as("q")
-.join(JoinRelType.INNER);
-return builder.call(SqlStdOperatorTable.CASE,
-builder.call(SqlStdOperatorTable.EQUALS,
-builder.field("q", "c"), builder.literal(0)),
-builder.literal(false),
-builder.call(SqlStdOperatorTable.IS_TRUE,
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m"))),
-builder.literal(true),
-builder.call(SqlStdOperatorTable.GREATER_THAN,
-builder.field("q", "c"), builder.field("q", "d")),
-
e.rel.getCluster().getRexBuilder().makeNullLiteral(SqlTypeName.BOOLEAN),
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m")));
+assert (op == SqlStdOperatorTable.SOME_GE || op ==
SqlStdOperatorTable.SOME_LE
+|| op == SqlStdOperatorTable.SOME_LT || op ==
SqlStdOperatorTable.SOME_GT);
+if (variablesSet.isEmpty()) {
+ builder.push(e.rel)
+ .aggregate(builder.groupKey(),
+ op.comparisonKind == SqlKind.GREATER_THAN
+ || op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
+ ? builder.min("m", builder.field(0))
+ : builder.max("m", builder.field(0)),
+ builder.count(false, "c"),
+ builder.count(false, "d", builder.field(0)))
+ .as("q")
+ .join(JoinRelType.INNER);
+ return builder.call(SqlStdOperatorTable.CASE,
+ builder.call(SqlStdOperatorTable.EQUALS,
+ builder.field("q", "c"), builder.literal(0)),
Review comment:
In this case it should be safe to rely on aliases since rewrite will produce
only `c`, `d`, and `m`.
With correlated case plan can have an extra `group by ` but planner
appropriately rename the column.
e.g.
`explain cbo select * from t45 where i > ANY(select i from t45 t where t.c =
t45.c);`
generates
```
HiveProject(i=[$0], c=[$1])
HiveFilter(condition=[OR(AND(>($0, $2), IS NOT TRUE(OR(IS NULL($5), =($3,
0, AND(>($0, $2), IS NOT TRUE(OR(IS NULL($5), =($3, 0))), IS NOT TRUE(>($0,
$2)), IS NOT TRUE(>($3, $4])
HiveJoin(condition=[=($6, $1)], joinType=[left], algorithm=[none],
cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
HiveProject(i=[$0], c=[$1])
HiveTableScan(table=[[default, t45]], table:alias=[t45])
HiveProject(m=[$1], c=[$2], d=[$3], alwaysTrue2115=[true], c0=[$0])
HiveAggregate(group=[{1}], m=[MIN($0)], c=[COUNT()], d=[COUNT($0)])
HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[default, t45]], table:alias=[t])
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271227)
Time Spent: 1h 10m (was: 1h)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter:
[jira] [Updated] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21921: --- Attachment: HIVE-21921.5.patch > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch, HIVE-21921.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271226
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 21:09
Start Date: 02/Jul/19 21:09
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #693:
HIVE-21921: Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299674222
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -230,33 +228,59 @@ private RexNode rewriteScalar(RelMetadataQuery mq,
RexSubQuery e, Set
variablesSet,
HiveSubQRemoveRelBuilder builder) {
final SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
-assert(op == SqlStdOperatorTable.SOME_GE
-|| op == SqlStdOperatorTable.SOME_LE
-|| op == SqlStdOperatorTable.SOME_LT
-|| op == SqlStdOperatorTable.SOME_GT);
-builder.push(e.rel)
-.aggregate(builder.groupKey(),
-op.comparisonKind == SqlKind.GREATER_THAN
-|| op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
-? builder.min("m", builder.field(0))
-: builder.max("m", builder.field(0)),
-builder.count(false, "c"),
-builder.count(false, "d", builder.field(0)))
-.as("q")
-.join(JoinRelType.INNER);
-return builder.call(SqlStdOperatorTable.CASE,
-builder.call(SqlStdOperatorTable.EQUALS,
-builder.field("q", "c"), builder.literal(0)),
-builder.literal(false),
-builder.call(SqlStdOperatorTable.IS_TRUE,
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m"))),
-builder.literal(true),
-builder.call(SqlStdOperatorTable.GREATER_THAN,
-builder.field("q", "c"), builder.field("q", "d")),
-
e.rel.getCluster().getRexBuilder().makeNullLiteral(SqlTypeName.BOOLEAN),
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m")));
+assert (op == SqlStdOperatorTable.SOME_GE || op ==
SqlStdOperatorTable.SOME_LE
+|| op == SqlStdOperatorTable.SOME_LT || op ==
SqlStdOperatorTable.SOME_GT);
+if (variablesSet.isEmpty()) {
+ builder.push(e.rel)
+ .aggregate(builder.groupKey(),
+ op.comparisonKind == SqlKind.GREATER_THAN
+ || op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
+ ? builder.min("m", builder.field(0))
+ : builder.max("m", builder.field(0)),
+ builder.count(false, "c"),
+ builder.count(false, "d", builder.field(0)))
+ .as("q")
Review comment:
You mean alias `as q`? This is to be able to resolve fields like `m`, `c`
etc appropriately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271226)
Time Spent: 1h (was: 50m)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21945) Enable sorted dynamic partitioning optimization for materialized views with custom data organization
[ https://issues.apache.org/jira/browse/HIVE-21945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21945: -- > Enable sorted dynamic partitioning optimization for materialized views with > custom data organization > > > Key: HIVE-21945 > URL: https://issues.apache.org/jira/browse/HIVE-21945 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > After implementing HIVE-18842, we need to extend the optimizer to work with > partitioned materialized views that are created with custom data > organization, i.e., using CLUSTERED, DISTRIBUTED, or SORTED. Currently, > optimization bails out when the materialized view is partitioned and either > CLUSTERED, DISTRIBUTED, or SORTED. > In particular, we will need to combine the RS operator introduced by the > translation of these clauses with the new RS needed to distribute and sort > the data based on the dynamic partition values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877296#comment-16877296
]
Jesus Camacho Rodriguez edited comment on HIVE-18842 at 7/2/19 8:51 PM:
[~vgarg], can you review this patch?
https://github.com/apache/hive/pull/696
Thanks
was (Author: jcamachorodriguez):
[~vgarg], can you review this patch? Thanks
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-18842:
--
Labels: pull-request-available (was: )
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877296#comment-16877296
]
Jesus Camacho Rodriguez commented on HIVE-18842:
[~vgarg], can you review this patch? Thanks
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?focusedWorklogId=271198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271198
]
ASF GitHub Bot logged work on HIVE-18842:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 20:50
Start Date: 02/Jul/19 20:50
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #696: HIVE-18842
URL: https://github.com/apache/hive/pull/696
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271198)
Time Spent: 10m
Remaining Estimate: 0h
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18842:
---
Attachment: HIVE-18842.01.patch
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18842.01.patch, HIVE-18842.patch
>
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON/SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | ( [DISTRIBUTED ON (col_name, ...)] [SORTED
> ON (col_name, ...)] ) ] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21910) Multiple target location generation in HostAffinitySplitLocationProvider
[
https://issues.apache.org/jira/browse/HIVE-21910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877294#comment-16877294
]
Hive QA commented on HIVE-21910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch llap-tez passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 41 unchanged - 1
fixed = 41 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 2253 unchanged - 1 fixed = 2254
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Null passed for non-null parameter of new java.util.HashSet(Collection)
in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List,
boolean, int) Method invoked at HostAffinitySplitLocationProvider.java:of new
java.util.HashSet(Collection) in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List,
boolean, int) Method invoked at HostAffinitySplitLocationProvider.java:[line
68] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17819/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17819/yetus/new-findbugs-ql.html
|
| modules | C: common llap-tez ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17819/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Multiple target location generation in HostAffinitySplitLocationProvider
>
[jira] [Updated] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21909: --- Status: Patch Available (was: Open) > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877276#comment-16877276
]
Hive QA commented on HIVE-18842:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973398/HIVE-18842.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 379 failed/errored test(s), 16385 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=244)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_orc]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[approx_distinct]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_2]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin1]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin3]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_6]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_limit]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_udaf]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_udaf_stats_opt]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_views]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_semijoin]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_subq_in] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_udf_udaf] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_views] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cluster] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_access_stats]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_semijoin]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer3]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[count] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_empty_source]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[database] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[drop_partition_with_stats]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dst] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_semijoin_reduction_sw2]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extrapolate_part_stats_partial_ndv]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_id2]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_resolution]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32_lessSize]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown3]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_notequalns]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_decimal]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_emit_interval]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hint]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_partitioned_2]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_partitioned_3]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rebuild]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rewrite_10]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rewrite_1]
(batchId=68)
[jira] [Commented] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877254#comment-16877254
]
Hive QA commented on HIVE-18842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 9 new + 708 unchanged - 1
fixed = 717 total (was 709) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
15s{color} | {color:red} ql generated 2 new + 2253 unchanged - 1 fixed = 2255
total (was 2254) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to LA29_5 in
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At
HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At HiveLexer.java:[line 12954] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA242 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48940-48953] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17818/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17818/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17818/yetus/new-findbugs-ql.html
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17818/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CLUSTERED ON/DISTRIBUTED ON/SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho
[jira] [Commented] (HIVE-21911) Pluggable LlapMetricsListener on Tez side to disable / resize Daemons
[
https://issues.apache.org/jira/browse/HIVE-21911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877227#comment-16877227
]
Hive QA commented on HIVE-21911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973400/HIVE-21911.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16361 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17817/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17817/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17817/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973400 - PreCommit-HIVE-Build
> Pluggable LlapMetricsListener on Tez side to disable / resize Daemons
> -
>
> Key: HIVE-21911
> URL: https://issues.apache.org/jira/browse/HIVE-21911
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21911.2.patch, HIVE-21911.3.patch, HIVE-21911.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We need to have a way to plug in different listeners which act upon the
> LlapDaemon statistics.
> This listener should be able to disable / resize the LlapDaemons based on
> health data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21911) Pluggable LlapMetricsListener on Tez side to disable / resize Daemons
[
https://issues.apache.org/jira/browse/HIVE-21911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877176#comment-16877176
]
Hive QA commented on HIVE-21911:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17817/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common llap-tez U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17817/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Pluggable LlapMetricsListener on Tez side to disable / resize Daemons
> -
>
> Key: HIVE-21911
> URL: https://issues.apache.org/jira/browse/HIVE-21911
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Tez
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21911.2.patch, HIVE-21911.3.patch, HIVE-21911.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We need to have a way to plug in different listeners which act upon the
> LlapDaemon statistics.
> This listener should be able to disable / resize the LlapDaemons based on
> health data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877158#comment-16877158
]
Hive QA commented on HIVE-21874:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973395/HIVE-21874.06.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16617 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17816/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17816/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17816/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973395 - PreCommit-HIVE-Build
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch,
> HIVE-21874.03.patch, HIVE-21874.04.patch, HIVE-21874.05.patch,
> HIVE-21874.06.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271021
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 16:40
Start Date: 02/Jul/19 16:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #693: HIVE-21921:
Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299576240
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -230,33 +228,59 @@ private RexNode rewriteScalar(RelMetadataQuery mq,
RexSubQuery e, Set
variablesSet,
HiveSubQRemoveRelBuilder builder) {
final SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
-assert(op == SqlStdOperatorTable.SOME_GE
-|| op == SqlStdOperatorTable.SOME_LE
-|| op == SqlStdOperatorTable.SOME_LT
-|| op == SqlStdOperatorTable.SOME_GT);
-builder.push(e.rel)
-.aggregate(builder.groupKey(),
-op.comparisonKind == SqlKind.GREATER_THAN
-|| op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
-? builder.min("m", builder.field(0))
-: builder.max("m", builder.field(0)),
-builder.count(false, "c"),
-builder.count(false, "d", builder.field(0)))
-.as("q")
-.join(JoinRelType.INNER);
-return builder.call(SqlStdOperatorTable.CASE,
-builder.call(SqlStdOperatorTable.EQUALS,
-builder.field("q", "c"), builder.literal(0)),
-builder.literal(false),
-builder.call(SqlStdOperatorTable.IS_TRUE,
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m"))),
-builder.literal(true),
-builder.call(SqlStdOperatorTable.GREATER_THAN,
-builder.field("q", "c"), builder.field("q", "d")),
-
e.rel.getCluster().getRexBuilder().makeNullLiteral(SqlTypeName.BOOLEAN),
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m")));
+assert (op == SqlStdOperatorTable.SOME_GE || op ==
SqlStdOperatorTable.SOME_LE
+|| op == SqlStdOperatorTable.SOME_LT || op ==
SqlStdOperatorTable.SOME_GT);
+if (variablesSet.isEmpty()) {
+ builder.push(e.rel)
+ .aggregate(builder.groupKey(),
+ op.comparisonKind == SqlKind.GREATER_THAN
+ || op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
+ ? builder.min("m", builder.field(0))
+ : builder.max("m", builder.field(0)),
+ builder.count(false, "c"),
+ builder.count(false, "d", builder.field(0)))
+ .as("q")
+ .join(JoinRelType.INNER);
+ return builder.call(SqlStdOperatorTable.CASE,
+ builder.call(SqlStdOperatorTable.EQUALS,
+ builder.field("q", "c"), builder.literal(0)),
Review comment:
I think we should use positions to create the field references since I am
not sure how much we can rely on aliases. For instance, what would happen if
there is a group key with an alias 'c'? Wouldn't that cause a collision with
the name for the count column?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271021)
Time Spent: 40m (was: 0.5h)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271020
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 16:40
Start Date: 02/Jul/19 16:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #693: HIVE-21921:
Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299574261
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -230,33 +228,59 @@ private RexNode rewriteScalar(RelMetadataQuery mq,
RexSubQuery e, Set
variablesSet,
HiveSubQRemoveRelBuilder builder) {
final SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
-assert(op == SqlStdOperatorTable.SOME_GE
-|| op == SqlStdOperatorTable.SOME_LE
-|| op == SqlStdOperatorTable.SOME_LT
-|| op == SqlStdOperatorTable.SOME_GT);
-builder.push(e.rel)
-.aggregate(builder.groupKey(),
-op.comparisonKind == SqlKind.GREATER_THAN
-|| op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
-? builder.min("m", builder.field(0))
-: builder.max("m", builder.field(0)),
-builder.count(false, "c"),
-builder.count(false, "d", builder.field(0)))
-.as("q")
-.join(JoinRelType.INNER);
-return builder.call(SqlStdOperatorTable.CASE,
-builder.call(SqlStdOperatorTable.EQUALS,
-builder.field("q", "c"), builder.literal(0)),
-builder.literal(false),
-builder.call(SqlStdOperatorTable.IS_TRUE,
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m"))),
-builder.literal(true),
-builder.call(SqlStdOperatorTable.GREATER_THAN,
-builder.field("q", "c"), builder.field("q", "d")),
-
e.rel.getCluster().getRexBuilder().makeNullLiteral(SqlTypeName.BOOLEAN),
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m")));
+assert (op == SqlStdOperatorTable.SOME_GE || op ==
SqlStdOperatorTable.SOME_LE
+|| op == SqlStdOperatorTable.SOME_LT || op ==
SqlStdOperatorTable.SOME_GT);
+if (variablesSet.isEmpty()) {
+ builder.push(e.rel)
+ .aggregate(builder.groupKey(),
+ op.comparisonKind == SqlKind.GREATER_THAN
+ || op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
+ ? builder.min("m", builder.field(0))
+ : builder.max("m", builder.field(0)),
+ builder.count(false, "c"),
+ builder.count(false, "d", builder.field(0)))
+ .as("q")
Review comment:
Is this line really needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271020)
Time Spent: 0.5h (was: 20m)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271022
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 16:40
Start Date: 02/Jul/19 16:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #693: HIVE-21921:
Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299577038
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -592,6 +616,20 @@ public static RexSubQuery find(RexNode node) {
}
}
+public static void subqueryRestriction(RelNode relNode) {
+ if(relNode instanceof HiveAggregate) {
+HiveAggregate aggregate = (HiveAggregate)relNode;
+if(!aggregate.getAggCallList().isEmpty() &&
aggregate.getGroupSet().isEmpty()) {
+ throw new CalciteSubqueryRuntimeException("Subquery rewrite:
Aggregate without group by is not allowed");
+}
+ } else if(relNode instanceof HiveProject
+ || relNode instanceof HiveFilter) {
Review comment:
nitpick. indentation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271022)
Time Spent: 50m (was: 40m)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271017
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 16:40
Start Date: 02/Jul/19 16:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #693: HIVE-21921:
Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299577708
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -230,33 +228,59 @@ private RexNode rewriteScalar(RelMetadataQuery mq,
RexSubQuery e, Set
variablesSet,
HiveSubQRemoveRelBuilder builder) {
final SqlQuantifyOperator op = (SqlQuantifyOperator) e.op;
-assert(op == SqlStdOperatorTable.SOME_GE
-|| op == SqlStdOperatorTable.SOME_LE
-|| op == SqlStdOperatorTable.SOME_LT
-|| op == SqlStdOperatorTable.SOME_GT);
-builder.push(e.rel)
-.aggregate(builder.groupKey(),
-op.comparisonKind == SqlKind.GREATER_THAN
-|| op.comparisonKind == SqlKind.GREATER_THAN_OR_EQUAL
-? builder.min("m", builder.field(0))
-: builder.max("m", builder.field(0)),
-builder.count(false, "c"),
-builder.count(false, "d", builder.field(0)))
-.as("q")
-.join(JoinRelType.INNER);
-return builder.call(SqlStdOperatorTable.CASE,
-builder.call(SqlStdOperatorTable.EQUALS,
-builder.field("q", "c"), builder.literal(0)),
-builder.literal(false),
-builder.call(SqlStdOperatorTable.IS_TRUE,
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m"))),
-builder.literal(true),
-builder.call(SqlStdOperatorTable.GREATER_THAN,
-builder.field("q", "c"), builder.field("q", "d")),
-
e.rel.getCluster().getRexBuilder().makeNullLiteral(SqlTypeName.BOOLEAN),
-builder.call(RelOptUtil.op(op.comparisonKind, null),
-e.operands.get(0), builder.field("q", "m")));
+assert (op == SqlStdOperatorTable.SOME_GE || op ==
SqlStdOperatorTable.SOME_LE
+|| op == SqlStdOperatorTable.SOME_LT || op ==
SqlStdOperatorTable.SOME_GT);
+if (variablesSet.isEmpty()) {
Review comment:
Can we add comments below with the main operators we are creating (or at
least an intuition on the rewriting)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271017)
Time Spent: 20m (was: 10m)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[
https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271019
]
ASF GitHub Bot logged work on HIVE-21921:
-
Author: ASF GitHub Bot
Created on: 02/Jul/19 16:40
Start Date: 02/Jul/19 16:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #693: HIVE-21921:
Support for correlated quantified predicates
URL: https://github.com/apache/hive/pull/693#discussion_r299578022
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
##
@@ -592,6 +616,20 @@ public static RexSubQuery find(RexNode node) {
}
}
+public static void subqueryRestriction(RelNode relNode) {
+ if(relNode instanceof HiveAggregate) {
+HiveAggregate aggregate = (HiveAggregate)relNode;
+if(!aggregate.getAggCallList().isEmpty() &&
aggregate.getGroupSet().isEmpty()) {
+ throw new CalciteSubqueryRuntimeException("Subquery rewrite:
Aggregate without group by is not allowed");
+}
+ } else if(relNode instanceof HiveProject
+ || relNode instanceof HiveFilter) {
+subqueryRestriction(relNode.getInput(0));
+ } else {
+return;
Review comment:
This is not needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 271019)
> Support for correlated quantified predicates
>
>
> Key: HIVE-21921
> URL: https://issues.apache.org/jira/browse/HIVE-21921
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch,
> HIVE-21921.3.patch, HIVE-21921.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21921) Support for correlated quantified predicates
[ https://issues.apache.org/jira/browse/HIVE-21921?focusedWorklogId=271018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271018 ] ASF GitHub Bot logged work on HIVE-21921: - Author: ASF GitHub Bot Created on: 02/Jul/19 16:40 Start Date: 02/Jul/19 16:40 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #693: HIVE-21921: Support for correlated quantified predicates URL: https://github.com/apache/hive/pull/693#discussion_r299573284 ## File path: ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java ## @@ -55,10 +55,8 @@ import java.util.Set; import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories; -import org.apache.hadoop.hive.ql.optimizer.calcite.HiveSubQRemoveRelBuilder; -import org.apache.hadoop.hive.ql.optimizer.calcite.SubqueryConf; -import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter; +import org.apache.hadoop.hive.ql.optimizer.calcite.*; Review comment: Can we expand the imports back? Same for other class below. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271018) Time Spent: 20m (was: 10m) > Support for correlated quantified predicates > > > Key: HIVE-21921 > URL: https://issues.apache.org/jira/browse/HIVE-21921 > Project: Hive > Issue Type: New Feature > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21921.1.patch, HIVE-21921.2.patch, > HIVE-21921.3.patch, HIVE-21921.4.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877123#comment-16877123 ] Antal Sinkovits commented on HIVE-21909: [~odraese] [~pvary] could you please review. Thanks. > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?focusedWorklogId=271012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271012 ] ASF GitHub Bot logged work on HIVE-21909: - Author: ASF GitHub Bot Created on: 02/Jul/19 16:30 Start Date: 02/Jul/19 16:30 Worklog Time Spent: 10m Work Description: asinkovits commented on pull request #695: HIVE-21909: Publish the LLAP Daemon capacity through ZooKeeper and ho… URL: https://github.com/apache/hive/pull/695 …nor the capacity when scheduling new tasks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271012) Time Spent: 10m Remaining Estimate: 0h > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21909: -- Labels: pull-request-available (was: ) > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch > > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21874) Implement add partitions related methods on temporary table
[
https://issues.apache.org/jira/browse/HIVE-21874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877119#comment-16877119
]
Hive QA commented on HIVE-21874:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 15 unchanged - 0 fixed
= 16 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17816/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17816/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17816/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement add partitions related methods on temporary table
> ---
>
> Key: HIVE-21874
> URL: https://issues.apache.org/jira/browse/HIVE-21874
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21874.01.patch, HIVE-21874.02.patch,
> HIVE-21874.03.patch, HIVE-21874.04.patch, HIVE-21874.05.patch,
> HIVE-21874.06.patch
>
>
> IMetaStoreClient exposes the following add partition related methods:
> {code:java}
> Partition add_partition(Partition partition);
> int add_partitions(List partitions);
> int add_partitions_pspec(PartitionSpecProxy partitionSpec);
> List add_partitions(List partitions, boolean
> ifNotExists, boolean needResults);
> {code}
> These methods should be implemented in order to handle addition of partitions
> to temporary tables.
--
This message was sent by
[jira] [Updated] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21909: --- Attachment: HIVE-21909.01.patch > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-21909.01.patch > > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[ https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21923: Attachment: HIVE-21923.04.patch > Vectorized MapJoin may miss results when only the join key is selected > -- > > Key: HIVE-21923 > URL: https://issues.apache.org/jira/browse/HIVE-21923 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch, > HIVE-21923.03.patch, HIVE-21923.04.patch > > > HIVE-21189 have introduced some resultset changes > in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out > https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21437) Vectorization: Decimal64 division with integer columns
[
https://issues.apache.org/jira/browse/HIVE-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877081#comment-16877081
]
Hive QA commented on HIVE-21437:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973386/HIVE-21437.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16360 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17815/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17815/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17815/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973386 - PreCommit-HIVE-Build
> Vectorization: Decimal64 division with integer columns
> --
>
> Key: HIVE-21437
> URL: https://issues.apache.org/jira/browse/HIVE-21437
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21437.1.patch, HIVE-21437.2.patch,
> HIVE-21437.3.patch, HIVE-21437.6.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorizer fails for
> {code}
> CREATE temporary TABLE `catalog_Sales`(
> `cs_quantity` int,
> `cs_wholesale_cost` decimal(7,2),
> `cs_list_price` decimal(7,2),
> `cs_sales_price` decimal(7,2),
> `cs_ext_discount_amt` decimal(7,2),
> `cs_ext_sales_price` decimal(7,2),
> `cs_ext_wholesale_cost` decimal(7,2),
> `cs_ext_list_price` decimal(7,2),
> `cs_ext_tax` decimal(7,2),
> `cs_coupon_amt` decimal(7,2),
> `cs_ext_ship_cost` decimal(7,2),
> `cs_net_paid` decimal(7,2),
> `cs_net_paid_inc_tax` decimal(7,2),
> `cs_net_paid_inc_ship` decimal(7,2),
> `cs_net_paid_inc_ship_tax` decimal(7,2),
> `cs_net_profit` decimal(7,2))
> ;
> explain vectorization detail select maxcs_ext_list_price -
> cs_ext_wholesale_cost) - cs_ext_discount_amt) + cs_ext_sales_price) / 2) from
> catalog_sales;
> {code}
> {code}
> 'Map Vectorization:'
> 'enabled: true'
> 'enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true'
> 'inputFileFormats:
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> 'notVectorizedReason: SELECT operator: Could not instantiate
> DecimalColDivideDecimalScalar with arguments arguments: [21, 20, 22],
> argument classes: [Integer, Integer, Integer], exception:
> java.lang.IllegalArgumentException: java.lang.ClassCastException@63b56be0
> stack trace:
> sun.reflect.GeneratedConstructorAccessor.newInstance(Unknown
> Source),
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45),
> java.lang.reflect.Constructor.newInstance(Constructor.java:423),
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.instantiateExpression(VectorizationContext.java:2088),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4662),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4602),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4584),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5171),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:923),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:809),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:776),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:240),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2038),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:1990),
>
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1963),
> ...'
> 'vectorized: false'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Status: Patch Available (was: Open) Changes made in v4: 1) Separated the returned list for ExtendedTableInfo into RequiredRead and RequiredWrite Properties for ExtendedTableInfo to match that on the Table. 2) Added a check to only do CreateTable translation when not in test mode. This will avoid test failures. > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Status: Open (was: Patch Available) > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21838: - Attachment: HIVE-21838.4.patch > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21437) Vectorization: Decimal64 division with integer columns
[
https://issues.apache.org/jira/browse/HIVE-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877036#comment-16877036
]
Hive QA commented on HIVE-21437:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17815/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17815/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Decimal64 division with integer columns
> --
>
> Key: HIVE-21437
> URL: https://issues.apache.org/jira/browse/HIVE-21437
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21437.1.patch, HIVE-21437.2.patch,
> HIVE-21437.3.patch, HIVE-21437.6.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorizer fails for
> {code}
> CREATE temporary TABLE `catalog_Sales`(
> `cs_quantity` int,
> `cs_wholesale_cost` decimal(7,2),
> `cs_list_price` decimal(7,2),
> `cs_sales_price` decimal(7,2),
> `cs_ext_discount_amt` decimal(7,2),
> `cs_ext_sales_price` decimal(7,2),
> `cs_ext_wholesale_cost` decimal(7,2),
> `cs_ext_list_price` decimal(7,2),
> `cs_ext_tax` decimal(7,2),
> `cs_coupon_amt` decimal(7,2),
> `cs_ext_ship_cost` decimal(7,2),
> `cs_net_paid` decimal(7,2),
> `cs_net_paid_inc_tax` decimal(7,2),
> `cs_net_paid_inc_ship` decimal(7,2),
> `cs_net_paid_inc_ship_tax` decimal(7,2),
> `cs_net_profit` decimal(7,2))
> ;
> explain vectorization detail select maxcs_ext_list_price -
> cs_ext_wholesale_cost) - cs_ext_discount_amt) + cs_ext_sales_price) / 2) from
> catalog_sales;
> {code}
> {code}
> 'Map Vectorization:'
> 'enabled: true'
> 'enabledConditionsMet:
> hive.vectorized.use.vectorized.input.format IS true'
> 'inputFileFormats:
>
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21944: --- Status: Patch Available (was: In Progress) > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Attachments: HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-21944: --- Attachment: HIVE-21944.1.patch > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Attachments: HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21912) Implement DisablingDaemonStatisticsHandler
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21912: -- Attachment: HIVE-21912.wip-2.patch > Implement DisablingDaemonStatisticsHandler > -- > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21912.wip-2.patch, HIVE-21912.wip.patch > > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21912) Implement DisablingDaemonStatisticsHandler
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21912: -- Attachment: Brokerage Statement - - 201906.pdf > Implement DisablingDaemonStatisticsHandler > -- > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21912.wip.patch > > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21912) Implement DisablingDaemonStatisticsHandler
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21912: -- Attachment: (was: Brokerage Statement - - 201906.pdf) > Implement DisablingDaemonStatisticsHandler > -- > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21912.wip.patch > > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21912) Implement DisablingDaemonStatisticsHandler
[ https://issues.apache.org/jira/browse/HIVE-21912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-21912: - Assignee: Peter Vary > Implement DisablingDaemonStatisticsHandler > -- > > Key: HIVE-21912 > URL: https://issues.apache.org/jira/browse/HIVE-21912 > Project: Hive > Issue Type: Sub-task > Components: llap, Tez >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21912.wip.patch > > > We should implement a DaemonStatisticsHandler which: > * If a node average response time is bigger than 150% (configurable) of the > other nodes > * If the other nodes has enough empty executors to handle the requests > Then disables the limping node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-21944: -- > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21944 started by Ivan Suller. -- > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876980#comment-16876980
]
Hive QA commented on HIVE-21923:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 23m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} common: The patch generated 1 new + 1914 unchanged - 0
fixed = 1915 total (was 1914) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 16m
11s{color} | {color:red} root: The patch generated 1 new + 73200 unchanged - 0
fixed = 73201 total (was 73200) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17814/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17814/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17814/yetus/diff-checkstyle-root.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17814/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch,
> HIVE-21923.03.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500
--
This message was sent by
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876956#comment-16876956
]
Hive QA commented on HIVE-21923:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973380/HIVE-21923.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16330 tests
executed
*Failed tests:*
{noformat}
TestReplAcrossInstancesWithJsonMessageFormat - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=151)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17814/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17814/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17814/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973380 - PreCommit-HIVE-Build
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch,
> HIVE-21923.03.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21943) Why do we not have audit log in HiveServer2?
[ https://issues.apache.org/jira/browse/HIVE-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HIVE-21943: -- Description: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would connect with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? We could add much infomations in the audit log of HS2, like username, ipAddress, queryId, queryStr, etc. was:Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would connect with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? > Why do we not have audit log in HiveServer2? > > > Key: HIVE-21943 > URL: https://issues.apache.org/jira/browse/HIVE-21943 > Project: Hive > Issue Type: Improvement >Reporter: Hui An >Priority: Major > > Currently, we have audit log in the metastore( see > [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides > us useful info, like metadata operations, ip address, etc. But when we use > HS2 to connect HMS, all operations would connect with HS2 ip address, not > actual user's ip address. This could cause difficult problems when we want to > locate user's ip address. So I am thinking if we could add audit log in the > HS2 too? > We could add much infomations in the audit log of HS2, like username, > ipAddress, queryId, queryStr, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21943) Why do we not have audit log in HiveServer2?
[ https://issues.apache.org/jira/browse/HIVE-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HIVE-21943: -- Description: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would assocate with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? We could add much infomations in the audit log of HS2, like username, ipAddress, queryId, queryStr, etc. was: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would connect with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? We could add much infomations in the audit log of HS2, like username, ipAddress, queryId, queryStr, etc. > Why do we not have audit log in HiveServer2? > > > Key: HIVE-21943 > URL: https://issues.apache.org/jira/browse/HIVE-21943 > Project: Hive > Issue Type: Improvement >Reporter: Hui An >Priority: Major > > Currently, we have audit log in the metastore( see > [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides > us useful info, like metadata operations, ip address, etc. But when we use > HS2 to connect HMS, all operations would assocate with HS2 ip address, not > actual user's ip address. This could cause difficult problems when we want to > locate user's ip address. So I am thinking if we could add audit log in the > HS2 too? > We could add much infomations in the audit log of HS2, like username, > ipAddress, queryId, queryStr, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21943) Why do we not have audit log in HiveServer2?
[ https://issues.apache.org/jira/browse/HIVE-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HIVE-21943: -- Description: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would connect with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too? (was: Currently, we have audit log in the metastore( see [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides us useful info, like metadata operations, ip address, etc. But when we use HS2 to connect HMS, all operations would connect with HS2 ip address, not actual user's ip address. This could cause difficult problems when we want to locate user's ip address. So I am thinking if we could add audit log in the HS2 too?) > Why do we not have audit log in HiveServer2? > > > Key: HIVE-21943 > URL: https://issues.apache.org/jira/browse/HIVE-21943 > Project: Hive > Issue Type: Improvement >Reporter: Hui An >Priority: Major > > Currently, we have audit log in the metastore( see > [HIVE-1948|https://issues.apache.org/jira/browse/HIVE-1948]), which provides > us useful info, like metadata operations, ip address, etc. But when we use > HS2 to connect HMS, all operations would connect with HS2 ip address, not > actual user's ip address. This could cause difficult problems when we want to > locate user's ip address. So I am thinking if we could add audit log in the > HS2 too? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21887) Multiple implementations of PersistenceManager are on the classpath
[
https://issues.apache.org/jira/browse/HIVE-21887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876860#comment-16876860
]
Hive QA commented on HIVE-21887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973381/HIVE-21887.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16349 tests
executed
*Failed tests:*
{noformat}
TestCliDriverMethods - did not produce a TEST-*.xml file (likely timed out)
(batchId=202)
TestRCFileCat - did not produce a TEST-*.xml file (likely timed out)
(batchId=202)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17813/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17813/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17813/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973381 - PreCommit-HIVE-Build
> Multiple implementations of PersistenceManager are on the classpath
> ---
>
> Key: HIVE-21887
> URL: https://issues.apache.org/jira/browse/HIVE-21887
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21887.01.patch, HIVE-21887.01.patch,
> HIVE-21887.01.patch
>
>
> While writing some test I've just bumped into that PersistenceManager is not
> always AutoCloseable ; however when I was using it from the metastore it was.
> it turned out that:
> * 'standalone-metastore' uses: org.datanucleus:javax.jdo:3.2.0-m3
> * 'metastore' uses: javax.jdo:jdo-api:3.0.1
> the problem is that both dependency contains the jdo api; and sometimes
> jdo-api is earlier on the classpath
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21887) Multiple implementations of PersistenceManager are on the classpath
[
https://issues.apache.org/jira/browse/HIVE-21887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876843#comment-16876843
]
Hive QA commented on HIVE-21887:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17813/dev-support/hive-personality.sh
|
| git revision | master / 2d9e0e4 |
| Default Java | 1.8.0_111 |
| modules | C: metastore . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17813/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Multiple implementations of PersistenceManager are on the classpath
> ---
>
> Key: HIVE-21887
> URL: https://issues.apache.org/jira/browse/HIVE-21887
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21887.01.patch, HIVE-21887.01.patch,
> HIVE-21887.01.patch
>
>
> While writing some test I've just bumped into that PersistenceManager is not
> always AutoCloseable ; however when I was using it from the metastore it was.
> it turned out that:
> * 'standalone-metastore' uses: org.datanucleus:javax.jdo:3.2.0-m3
> * 'metastore' uses: javax.jdo:jdo-api:3.0.1
> the problem is that both dependency contains the jdo api; and sometimes
> jdo-api is earlier on the classpath
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
