[jira] [Commented] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577403#comment-16577403
]
Hive QA commented on HIVE-20366:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935265/HIVE-20366.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14877 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13179/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13179/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13179/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935265 - PreCommit-HIVE-Build
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | outputColumnNames: _col0, _col1, _col3, _col4, _col5,
> _col6, _col8 |
> | input vertices:|
> | 1 Map 14 |
> | Statistics: Num rows: 10282477384 Data size: 534184867432
> Basic stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator|
> | predicate: _col8 is null (type: boolean) |
> | * Statistics: Num rows: 1* Data size: 52 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577398#comment-16577398
]
Hive QA commented on HIVE-20366:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 3 new + 23 unchanged - 0 fixed
= 26 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13179/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13179/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13179/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> |
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Status: Patch Available (was: In Progress) > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch, HIVE-19924.13.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Attachment: HIVE-19924.13.patch > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch, HIVE-19924.13.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Status: In Progress (was: Patch Available) > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch, HIVE-19924.13.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577395#comment-16577395
]
Hive QA commented on HIVE-20368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935264/HIVE-20368.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13178/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13178/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13178/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935264 - PreCommit-HIVE-Build
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch, HIVE-20368.2.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577386#comment-16577386
]
Hive QA commented on HIVE-20368:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13178/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13178/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch, HIVE-20368.2.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577382#comment-16577382
]
Hive QA commented on HIVE-20367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935263/HIVE-20367.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13177/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13177/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13177/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935263 - PreCommit-HIVE-Build
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch,
> HIVE-20367.03.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577380#comment-16577380
]
Vineet Garg commented on HIVE-20366:
The idea I am experimenting with is to use difference of NDVs to estimate
cardinality and then subtract that from estimated join cardinality to estimate
number of mismatched rows which are then used to estimate number of nulls.
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | outputColumnNames: _col0, _col1, _col3, _col4, _col5,
> _col6, _col8 |
> | input vertices:|
> | 1 Map 14 |
> | Statistics: Num rows: 10282477384 Data size: 534184867432
> Basic stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator|
> | predicate: _col8 is null (type: boolean) |
> | * Statistics: Num rows: 1* Data size: 52 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20366:
---
Status: Patch Available (was: Open)
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | outputColumnNames: _col0, _col1, _col3, _col4, _col5,
> _col6, _col8 |
> | input vertices:|
> | 1 Map 14 |
> | Statistics: Num rows: 10282477384 Data size: 534184867432
> Basic stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator|
> | predicate: _col8 is null (type: boolean) |
> | * Statistics: Num rows: 1* Data size: 52 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20366:
---
Status: Open (was: Patch Available)
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | outputColumnNames: _col0, _col1, _col3, _col4, _col5,
> _col6, _col8 |
> | input vertices:|
> | 1 Map 14 |
> | Statistics: Num rows: 10282477384 Data size: 534184867432
> Basic stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator|
> | predicate: _col8 is null (type: boolean) |
> | * Statistics: Num rows: 1* Data size: 52 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20366) TPC-DS query78 stats estimates are off for is null filter
[
https://issues.apache.org/jira/browse/HIVE-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20366:
---
Attachment: HIVE-20366.2.patch
> TPC-DS query78 stats estimates are off for is null filter
> -
>
> Key: HIVE-20366
> URL: https://issues.apache.org/jira/browse/HIVE-20366
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20366.1.patch, HIVE-20366.2.patch
>
>
> In Query 78, there is Left outer join between fact table combos: stores_sales
> LOJ store_returns, catalog_sales LOJ catalog_returns and web_sales LOJ
> web_returns. Each of these joins estimates only a single row and the result
> is BROADCAST and causes hash table memory errors
> {code}
> Reducer 12 |
> | Execution mode: vectorized, llap |
> | Reduce Operator Tree: |
> ++
> | Explain |
> ++
> | Map Join Operator|
> | condition map: |
> | Left Outer Join 0 to 1|
> | keys: |
> | 0 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | 1 KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1
> (type: bigint) |
> | outputColumnNames: _col0, _col1, _col3, _col4, _col5,
> _col6, _col8 |
> | input vertices:|
> | 1 Map 14 |
> | Statistics: Num rows: 10282477384 Data size: 534184867432
> Basic stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator|
> | predicate: _col8 is null (type: boolean) |
> | * Statistics: Num rows: 1* Data size: 52 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577375#comment-16577375
]
Hive QA commented on HIVE-20367:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 28 new + 436 unchanged - 2
fixed = 464 total (was 438) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
29s{color} | {color:red} ql generated 1 new + 2306 unchanged - 0 fixed = 2307
total (was 2306) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Integral division result cast to double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch)
At VectorPTFEvaluatorStreamingLongAvg.java:double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch)
At VectorPTFEvaluatorStreamingLongAvg.java:[line 82] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13177/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13177/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13177/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13177/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch,
> HIVE-20367.03.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-20368:
--
Attachment: HIVE-20368.2.patch
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch, HIVE-20368.2.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Status: Patch Available (was: In Progress)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch,
> HIVE-20367.03.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Attachment: HIVE-20367.03.patch
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch,
> HIVE-20367.03.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Status: In Progress (was: Patch Available)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17040) Join elimination in the presence of FK relationship
[
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577355#comment-16577355
]
Hive QA commented on HIVE-17040:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 10 new + 172 unchanged - 4
fixed = 182 total (was 176) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
10s{color} | {color:red} ql generated 1 new + 2306 unchanged - 0 fixed = 2307
total (was 2306) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to p1 in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule$EquivalenceClasses.addEquivalenceClass(RexTableInputRef,
RexTableInputRef) At
HiveJoinConstraintsRule.java:org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule$EquivalenceClasses.addEquivalenceClass(RexTableInputRef,
RexTableInputRef) At HiveJoinConstraintsRule.java:[line 391] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13176/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13176/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13176/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13176/yetus/new-findbugs-ql.html
|
| modules | C: common . itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13176/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
> Issue Type: Sub-task
> Components: Logical
[jira] [Commented] (HIVE-17040) Join elimination in the presence of FK relationship
[
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577353#comment-16577353
]
Hive QA commented on HIVE-17040:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935258/HIVE-17040.01.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 14880 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=264)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguitycheck]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_2]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_1]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_2]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_3]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_oneskew_1]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_oneskew_2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[macro] (batchId=4)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_dummy]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db]
(batchId=174)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[list_bucket_dml_2]
(batchId=114)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13176/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13176/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13176/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935258 - PreCommit-HIVE-Build
> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17040.01.patch, HIVE-17040.patch
>
>
> If the PK/UK table is not filtered, we can safely remove the join.
> A simple example:
> {code:sql}
> SELECT c_current_cdemo_sk
> FROM customer, customer_address
> ON c_current_addr_sk = ca_address_sk;
> {code}
> As a Calcite rule, we could implement this rewriting by 1) matching a Project
> on top of a Join operator, 2) checking that only columns from the FK are used
> in the Project, 3) checking that the join condition matches the FK - PK/UK
> relationship, 4) pulling all the predicates from the PK/UK side and checking
> that the input is not filtered, and 5) removing the join, possibly adding a
> IS NOT NULL condition on the join column from the FK side.
> If the PK/UK table is filtered, we should still transform the Join into a
> SemiJoin operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577324#comment-16577324
]
Hive QA commented on HIVE-20367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935244/HIVE-20367.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13175/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13175/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13175/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935244 - PreCommit-HIVE-Build
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577310#comment-16577310
]
Hive QA commented on HIVE-20367:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 28 new + 408 unchanged - 2
fixed = 436 total (was 410) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
37s{color} | {color:red} ql generated 1 new + 2306 unchanged - 0 fixed = 2307
total (was 2306) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Integral division result cast to double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch,
boolean) At VectorPTFEvaluatorStreamingLongAvg.java:double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch,
boolean) At VectorPTFEvaluatorStreamingLongAvg.java:[line 81] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13175/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13175/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13175/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13175/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17040) Join elimination in the presence of FK relationship
[
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17040:
---
Attachment: HIVE-17040.01.patch
> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17040.01.patch, HIVE-17040.patch
>
>
> If the PK/UK table is not filtered, we can safely remove the join.
> A simple example:
> {code:sql}
> SELECT c_current_cdemo_sk
> FROM customer, customer_address
> ON c_current_addr_sk = ca_address_sk;
> {code}
> As a Calcite rule, we could implement this rewriting by 1) matching a Project
> on top of a Join operator, 2) checking that only columns from the FK are used
> in the Project, 3) checking that the join condition matches the FK - PK/UK
> relationship, 4) pulling all the predicates from the PK/UK side and checking
> that the input is not filtered, and 5) removing the join, possibly adding a
> IS NOT NULL condition on the join column from the FK side.
> If the PK/UK table is filtered, we should still transform the Join into a
> SemiJoin operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Status: Patch Available (was: In Progress)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20329) Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
[
https://issues.apache.org/jira/browse/HIVE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577298#comment-16577298
]
Hive QA commented on HIVE-20329:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935253/HIVE-20329.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13174/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13174/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13174/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935253 - PreCommit-HIVE-Build
> Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
> --
>
> Key: HIVE-20329
> URL: https://issues.apache.org/jira/browse/HIVE-20329
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20329.01.patch
>
>
> The task created in the previous iterations of the load are not delinked and
> thus causing heap memory usage issue. need to delink the tasks to avoid OOM
> error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20329) Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
[
https://issues.apache.org/jira/browse/HIVE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577285#comment-16577285
]
Hive QA commented on HIVE-20329:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13174/dev-support/hive-personality.sh
|
| git revision | master / 28b24db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13174/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
> --
>
> Key: HIVE-20329
> URL: https://issues.apache.org/jira/browse/HIVE-20329
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20329.01.patch
>
>
> The task created in the previous iterations of the load are not delinked and
> thus causing heap memory usage issue. need to delink the tasks to avoid OOM
> error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20329) Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
[ https://issues.apache.org/jira/browse/HIVE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20329: --- Attachment: HIVE-20329.01.patch > Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error > -- > > Key: HIVE-20329 > URL: https://issues.apache.org/jira/browse/HIVE-20329 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20329.01.patch > > > The task created in the previous iterations of the load are not delinked and > thus causing heap memory usage issue. need to delink the tasks to avoid OOM > error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20329) Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error
[ https://issues.apache.org/jira/browse/HIVE-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20329: --- Status: Patch Available (was: Open) > Repl Scale Test : Running long running load (incr/bootstrap) causing OOM error > -- > > Key: HIVE-20329 > URL: https://issues.apache.org/jira/browse/HIVE-20329 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20329.01.patch > > > The task created in the previous iterations of the load are not delinked and > thus causing heap memory usage issue. need to delink the tasks to avoid OOM > error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20369) TestPreUpgradeTool not run by ptest
[ https://issues.apache.org/jira/browse/HIVE-20369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20369: - > TestPreUpgradeTool not run by ptest > --- > > Key: HIVE-20369 > URL: https://issues.apache.org/jira/browse/HIVE-20369 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > TestPreUpgradeTool is not showing up in ptest runs > probably because upgrade-acid module is disconnected from root pom > how does standalone-metastore work? it's also disconnected > also, hive-upgrade jar is not showing up in tar with mvn package -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19749) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-19749. --- Resolution: Fixed Fix Version/s: 4.0.0 > Acid V1 to V2 upgrade > - > > Key: HIVE-19749 > URL: https://issues.apache.org/jira/browse/HIVE-19749 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 4.0.0 > > > umbrella jira -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch, HIVE-19800.07.patch, HIVE-19800.08.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[
https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577260#comment-16577260
]
Hive QA commented on HIVE-19800:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} upgrade-acid in master has 1 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} root: The patch generated 0 new + 0 unchanged - 422
fixed = 0 total (was 422) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch packaging passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 47 new + 795 unchanged - 2
fixed = 842 total (was 797) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} upgrade-acid: The patch generated 48 new + 243
unchanged - 179 fixed = 291 total (was 422) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 291 new
+ 0 unchanged - 0 fixed = 291 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
23s{color} | {color:red} ql generated 1 new + 2305 unchanged - 0 fixed = 2306
total (was 2305) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.util.UpgradeTool.createScript(List, String,
String):in org.apache.hadoop.hive.ql.util.UpgradeTool.createScript(List,
String, String): new java.io.FileWriter(String) At UpgradeTool.java:[line 640]
|
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux
[jira] [Updated] (HIVE-20364) Update default for hive.map.aggr.hash.min.reduction
[ https://issues.apache.org/jira/browse/HIVE-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20364: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. > Update default for hive.map.aggr.hash.min.reduction > --- > > Key: HIVE-20364 > URL: https://issues.apache.org/jira/browse/HIVE-20364 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Nita Dembla >Assignee: Ashutosh Chauhan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20364.patch > > > Default value is 0.5 Lets update it to 0.99 > In average case its a trade-off between cpu vs network. Erring on side of CPU > is better since perf loss caused by network is usually larger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[
https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577259#comment-16577259
]
Hive QA commented on HIVE-19800:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935250/HIVE-19800.08.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13173/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13173/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13173/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935250 - PreCommit-HIVE-Build
> Create separate submodules for pre and post upgrade and add rename file logic
> -
>
> Key: HIVE-19800
> URL: https://issues.apache.org/jira/browse/HIVE-19800
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch,
> HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch,
> HIVE-19800.06.patch, HIVE-19800.07.patch, HIVE-19800.08.patch
>
>
> this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't
> landed yet
> this includes file rename logic and HIVE-19750 since it hasn't landed yet
> either
>
> cc [~jdere]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Attachment: HIVE-19800.08.patch > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch, HIVE-19800.07.patch, HIVE-19800.08.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[
https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577218#comment-16577218
]
Hive QA commented on HIVE-19800:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935248/HIVE-19800.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13172/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13172/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-08-11 15:50:29.426
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-13172/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-08-11 15:50:29.430
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 380946c HIVE-20361: ReplDumpTaskTest is failing (Sankar
Hariappan, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 380946c HIVE-20361: ReplDumpTaskTest is failing (Sankar
Hariappan, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-08-11 15:50:30.391
+ rm -rf ../yetus_PreCommit-HIVE-Build-13172
+ mkdir ../yetus_PreCommit-HIVE-Build-13172
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-13172
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-13172/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive: Could not resolve dependencies
for project org.apache.hive:hive:pom:4.0.0-SNAPSHOT: Could not find artifact
org.apache.hive:hive-upgrade-acid:jar:4.0.0-SNAPSHOT in apache.snapshots
(https://repository.apache.org/snapshots) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-13172
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935248 - PreCommit-HIVE-Build
> Create separate submodules for pre and post upgrade and add rename file logic
> -
>
> Key: HIVE-19800
> URL: https://issues.apache.org/jira/browse/HIVE-19800
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch,
> HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch,
> HIVE-19800.06.patch, HIVE-19800.07.patch
>
>
> this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't
> landed yet
>
[jira] [Commented] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577216#comment-16577216
]
Hive QA commented on HIVE-20368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935245/HIVE-20368.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=166)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13171/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13171/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13171/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935245 - PreCommit-HIVE-Build
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Status: Patch Available (was: Open) > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch, HIVE-19800.07.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Attachment: HIVE-19800.07.patch > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch, HIVE-19800.07.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19800: -- Status: Open (was: Patch Available) > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch, HIVE-19800.07.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577212#comment-16577212
]
Hive QA commented on HIVE-20368:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13171/dev-support/hive-personality.sh
|
| git revision | master / 380946c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13171/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-20368:
--
Labels: pull-request-available (was: )
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577198#comment-16577198
]
ASF GitHub Bot commented on HIVE-20368:
---
GitHub user pudidic opened a pull request:
https://github.com/apache/hive/pull/414
HIVE-20368: Remove VectorTopNKeyOperator lock (Teddy Choi)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pudidic/hive HIVE-20368
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/414.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #414
commit 149a613dcc94f90423ed1be2090ee87df8266a46
Author: Teddy Choi
Date: 2018-08-11T14:29:52Z
HIVE-20368: Remove VectorTopNKeyOperator lock (Teddy Choi)
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-20368:
--
Status: Patch Available (was: Open)
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-20368:
--
Attachment: HIVE-20368.1.patch
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-20368.1.patch
>
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20368) Remove VectorTopNKeyOperator lock
[
https://issues.apache.org/jira/browse/HIVE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi reassigned HIVE-20368:
-
> Remove VectorTopNKeyOperator lock
> -
>
> Key: HIVE-20368
> URL: https://issues.apache.org/jira/browse/HIVE-20368
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
>
> VectorTopNKeyOperator has a lock in line 199 as following.
> {code:java}
> priorityQueue.offer(WritableUtils.clone(keysWritable, getConfiguration()));
> {code}
> WritableUtils.clone calls Confgiruation.getClassByNameOrNull that has
> synchronized block. So it needs to run without locks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Status: In Progress (was: Patch Available)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Attachment: HIVE-20367.02.patch
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch, HIVE-20367.02.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577178#comment-16577178
]
Hive QA commented on HIVE-20367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935241/HIVE-20367.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ptf]
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf] (batchId=117)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.org.apache.hadoop.hive.ql.parse.TestCopyUtils
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13170/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13170/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13170/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935241 - PreCommit-HIVE-Build
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577171#comment-16577171
]
Hive QA commented on HIVE-20367:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 28 new + 408 unchanged - 2
fixed = 436 total (was 410) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
14s{color} | {color:red} ql generated 1 new + 2305 unchanged - 0 fixed = 2306
total (was 2305) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Integral division result cast to double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch,
boolean) At VectorPTFEvaluatorStreamingLongAvg.java:double or float in
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFEvaluatorStreamingLongAvg.evaluateGroupBatch(VectorizedRowBatch,
boolean) At VectorPTFEvaluatorStreamingLongAvg.java:[line 81] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13170/dev-support/hive-personality.sh
|
| git revision | master / 380946c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13170/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13170/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13170/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Status: Patch Available (was: Open)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Attachment: HIVE-20367.01.patch
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Attachment: (was: HIVE-20367.01.patch)
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19927) Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.
[ https://issues.apache.org/jira/browse/HIVE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577146#comment-16577146 ] Alisha Prabhu commented on HIVE-19927: -- Thanks for the quick fix [~sankarh] !! > Last Repl ID set by bootstrap dump is incorrect and may cause data loss if > have ACID/MM tables. > --- > > Key: HIVE-19927 > URL: https://issues.apache.org/jira/browse/HIVE-19927 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl, Transactions >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19927.01-branch-3.patch, HIVE-19927.01.patch, > HIVE-19927.02.patch, HIVE-19927.03.patch, HIVE-19927.04.patch > > > During bootstrap dump of ACID tables, let's consider the below sequence. > - Current session (REPL DUMP), Open txn (Txn1) - Event-10 > - Another session (Session-2), Open txn (Txn2) - Event-11 > - Session-2 -> Insert data (T1.D1) to ACID table. - Event-12 > - Get lastReplId = last event ID logged. (Event-12) > - Session-2 -> Commit Txn (Txn2) - Event-13 > - Dump ACID tables based on validTxnList based on Txn1. --> This step skips > all the data written by txns > Txn1. So, T1.D1 will be missing. > - Commit Txn (Txn1) > - REPL LOAD from bootstrap dump will skip T1.D1. > - Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is > opened after Txn1. So, data T1.D1 will be lost for ever. > Proposed to capture the lastReplId of bootstrap before opening current txn > (Txn1) and store it in Driver context and use it for dump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20367:
Attachment: HIVE-20367.01.patch
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20367.01.patch
>
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20367) Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
[
https://issues.apache.org/jira/browse/HIVE-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline reassigned HIVE-20367:
---
> Vectorization: Support streaming for PTF AVG, MAX, MIN, SUM
> ---
>
> Key: HIVE-20367
> URL: https://issues.apache.org/jira/browse/HIVE-20367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> Add support for vectorizing PTF AVG, MAX, MIN, SUM when:
> {noformat}
> ROWS PRECEDING(MAX)~CURRENT
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19924) Tag distcp jobs run by Repl Load
[
https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577092#comment-16577092
]
Hive QA commented on HIVE-19924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935234/HIVE-19924.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14876 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=265)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=265)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13169/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13169/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13169/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935234 - PreCommit-HIVE-Build
> Tag distcp jobs run by Repl Load
>
>
> Key: HIVE-19924
> URL: https://issues.apache.org/jira/browse/HIVE-19924
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch,
> HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch,
> HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch,
> HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch,
> HIVE-19924.12.patch
>
>
> Add tags in jobconf for distcp related jobs started by replication. This will
> allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon
> issues a kill command.
> * one of the tags should definitely be the query_id that starts the job :
> With this flow beacon before retrying the bootstrap load, will issue a kill
> command to hs2 with the query id of the previous issued command. hs2 will
> then kill an running jobs on yarn tagged with the Query_id.
> * To get around the additional failure point as mentioned above. The jobs
> can be tagged with an additional unique tag_id provided by Beacon in the WITH
> clause in repl load command to be used to tag distcp jobs ). Enhance the kill
> api to take the tag as input and kill jobs associated with that tag. Problem
> here is how do we validate the association of the tag with a hive query id to
> make sure this api is not used to kill jobs run by other components, however
> we can provide this capability to only admins and should be ok in that case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19924) Tag distcp jobs run by Repl Load
[
https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577085#comment-16577085
]
Hive QA commented on HIVE-19924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 17 new + 230
unchanged - 0 fixed = 247 total (was 230) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 2 new + 306 unchanged - 13
fixed = 308 total (was 319) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13169/dev-support/hive-personality.sh
|
| git revision | master / 380946c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13169/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13169/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests/hive-unit ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13169/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tag distcp jobs run by Repl Load
>
>
> Key: HIVE-19924
> URL: https://issues.apache.org/jira/browse/HIVE-19924
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR, replication
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch,
> HIVE-19924.03.patch,
[jira] [Commented] (HIVE-17040) Join elimination in the presence of FK relationship
[
https://issues.apache.org/jira/browse/HIVE-17040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577084#comment-16577084
]
Gopal V commented on HIVE-17040:
Interesting to see what the query72 plan looks like with this change +
constraints.
> Join elimination in the presence of FK relationship
> ---
>
> Key: HIVE-17040
> URL: https://issues.apache.org/jira/browse/HIVE-17040
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-17040.patch
>
>
> If the PK/UK table is not filtered, we can safely remove the join.
> A simple example:
> {code:sql}
> SELECT c_current_cdemo_sk
> FROM customer, customer_address
> ON c_current_addr_sk = ca_address_sk;
> {code}
> As a Calcite rule, we could implement this rewriting by 1) matching a Project
> on top of a Join operator, 2) checking that only columns from the FK are used
> in the Project, 3) checking that the join condition matches the FK - PK/UK
> relationship, 4) pulling all the predicates from the PK/UK side and checking
> that the input is not filtered, and 5) removing the join, possibly adding a
> IS NOT NULL condition on the join column from the FK side.
> If the PK/UK table is filtered, we should still transform the Join into a
> SemiJoin operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Status: Patch Available (was: In Progress) moved kill query test to a llap based mini cluster test suite > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Status: In Progress (was: Patch Available) fixed defects related to kill query config > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Attachment: HIVE-19924.12.patch > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch, > HIVE-19924.09.patch, HIVE-19924.10.patch, HIVE-19924.11.patch, > HIVE-19924.12.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
