[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098695#comment-17098695
]
Hive QA commented on HIVE-23298:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001961/HIVE-23298.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17200 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22108/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22108/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22108/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001961 - PreCommit-HIVE-Build
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch,
> HIVE-23298.3.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098660#comment-17098660
]
Hive QA commented on HIVE-23298:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22108/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22108/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch,
> HIVE-23298.3.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23206) Project not defined correctly after reordering a join
[ https://issues.apache.org/jira/browse/HIVE-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23206: -- Attachment: HIVE-23206.5.patch > Project not defined correctly after reordering a join > - > > Key: HIVE-23206 > URL: https://issues.apache.org/jira/browse/HIVE-23206 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Steve Carlin >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23206.1.patch, HIVE-23206.2.patch, > HIVE-23206.3.patch, HIVE-23206.4.patch, HIVE-23206.5.patch > > > The following highlighted line seems to be incorrect in the test suite: > [https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89] > Note that the project takes all the columns from the table scan, yet it only > needs a couple of them. > I did some very small debugging on this. When I removed the > applyJoinOrderingTransform here: > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897] > ... the problem goes away. So presumably one of the rules in there is > causing the problem. > Here is a slightly simplified version of the query which has the same problem > (using the same tpc-ds database): > explain cbo with ws_wh as > (select ws1.ws_order_number > from web_sales ws1,web_returns wr2 > where ws1.ws_order_number = wr2.wr_order_number) > select > ws_order_number > from > web_sales ws1 > where > ws1.ws_order_number in (select wr_order_number > from web_returns,ws_wh > where wr_order_number = ws_wh.ws_order_number) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23206) Project not defined correctly after reordering a join
[ https://issues.apache.org/jira/browse/HIVE-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23206: -- Status: Open (was: Patch Available) > Project not defined correctly after reordering a join > - > > Key: HIVE-23206 > URL: https://issues.apache.org/jira/browse/HIVE-23206 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Steve Carlin >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23206.1.patch, HIVE-23206.2.patch, > HIVE-23206.3.patch, HIVE-23206.4.patch, HIVE-23206.5.patch > > > The following highlighted line seems to be incorrect in the test suite: > [https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89] > Note that the project takes all the columns from the table scan, yet it only > needs a couple of them. > I did some very small debugging on this. When I removed the > applyJoinOrderingTransform here: > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897] > ... the problem goes away. So presumably one of the rules in there is > causing the problem. > Here is a slightly simplified version of the query which has the same problem > (using the same tpc-ds database): > explain cbo with ws_wh as > (select ws1.ws_order_number > from web_sales ws1,web_returns wr2 > where ws1.ws_order_number = wr2.wr_order_number) > select > ws_order_number > from > web_sales ws1 > where > ws1.ws_order_number in (select wr_order_number > from web_returns,ws_wh > where wr_order_number = ws_wh.ws_order_number) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.39.patch > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch, > HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch, > HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch, > HIVE-21304.38.patch, HIVE-21304.38.patch, HIVE-21304.38.patch, > HIVE-21304.39.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23206) Project not defined correctly after reordering a join
[ https://issues.apache.org/jira/browse/HIVE-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23206: -- Status: Patch Available (was: Open) > Project not defined correctly after reordering a join > - > > Key: HIVE-23206 > URL: https://issues.apache.org/jira/browse/HIVE-23206 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Steve Carlin >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23206.1.patch, HIVE-23206.2.patch, > HIVE-23206.3.patch, HIVE-23206.4.patch, HIVE-23206.5.patch > > > The following highlighted line seems to be incorrect in the test suite: > [https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/perf/tez/cbo_query95.q.out#L89] > Note that the project takes all the columns from the table scan, yet it only > needs a couple of them. > I did some very small debugging on this. When I removed the > applyJoinOrderingTransform here: > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1897] > ... the problem goes away. So presumably one of the rules in there is > causing the problem. > Here is a slightly simplified version of the query which has the same problem > (using the same tpc-ds database): > explain cbo with ws_wh as > (select ws1.ws_order_number > from web_sales ws1,web_returns wr2 > where ws1.ws_order_number = wr2.wr_order_number) > select > ws_order_number > from > web_sales ws1 > where > ws1.ws_order_number in (select wr_order_number > from web_returns,ws_wh > where wr_order_number = ws_wh.ws_order_number) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Attachment: HIVE-23298.3.patch
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch,
> HIVE-23298.3.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Patch Available (was: Open)
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch,
> HIVE-23298.3.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Open (was: Patch Available)
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch,
> HIVE-23298.3.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22028) Clean up Add Partition
[
https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098633#comment-17098633
]
Hive QA commented on HIVE-22028:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001960/HIVE-22028.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17200 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22107/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22107/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22107/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001960 - PreCommit-HIVE-Build
> Clean up Add Partition
> --
>
> Key: HIVE-22028
> URL: https://issues.apache.org/jira/browse/HIVE-22028
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM,
> HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> AlterTableAddPartitionDesc should be immutable, like the rest of the desc
> classes. This can not be done 100% right now, as it requires the refactoring
> of the ImportSemanticAnalyzer, so the task will be finished only then.
> Add Partition logic should be moved from Hive.java to
> AlterTableAddPartitionOperation.java, only the metastore calls should remain
> in Hive.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22028) Clean up Add Partition
[
https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098626#comment-17098626
]
Hive QA commented on HIVE-22028:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
31s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 195 unchanged - 2
fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22107/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22107/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up Add Partition
> --
>
> Key: HIVE-22028
> URL: https://issues.apache.org/jira/browse/HIVE-22028
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM,
> HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> AlterTableAddPartitionDesc should be immutable, like the rest of the desc
> classes. This can not be done 100% right now, as it requires the refactoring
> of the ImportSemanticAnalyzer, so the task will be finished only then.
> Add Partition logic should be moved from Hive.java to
> AlterTableAddPartitionOperation.java, only the metastore calls should remain
> in Hive.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22028) Clean up Add Partition
[
https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098620#comment-17098620
]
Hive QA commented on HIVE-22028:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001959/HIVE-22028.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17200 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[7]
(batchId=141)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22106/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22106/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22106/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001959 - PreCommit-HIVE-Build
> Clean up Add Partition
> --
>
> Key: HIVE-22028
> URL: https://issues.apache.org/jira/browse/HIVE-22028
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM,
> HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> AlterTableAddPartitionDesc should be immutable, like the rest of the desc
> classes. This can not be done 100% right now, as it requires the refactoring
> of the ImportSemanticAnalyzer, so the task will be finished only then.
> Add Partition logic should be moved from Hive.java to
> AlterTableAddPartitionOperation.java, only the metastore calls should remain
> in Hive.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: (was: HIVE-22028.01.patch-ADDENDUM) > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM, > HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: HIVE-22028.01.patch-ADDENDUM > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM, > HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22028) Clean up Add Partition
[
https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098615#comment-17098615
]
Hive QA commented on HIVE-22028:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
33s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 195 unchanged - 2
fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22106/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22106/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up Add Partition
> --
>
> Key: HIVE-22028
> URL: https://issues.apache.org/jira/browse/HIVE-22028
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM,
> HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> AlterTableAddPartitionDesc should be immutable, like the rest of the desc
> classes. This can not be done 100% right now, as it requires the refactoring
> of the ImportSemanticAnalyzer, so the task will be finished only then.
> Add Partition logic should be moved from Hive.java to
> AlterTableAddPartitionOperation.java, only the metastore calls should remain
> in Hive.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: HIVE-22028.01.patch-ADDENDUM > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028.01.patch, HIVE-22028.01.patch-ADDENDUM, > HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: (was: HIVE-22028-ADDENDUM.01.patch) > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028.01.patch, HIVE-22028.02.patch, > HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: HIVE-22028-ADDENDUM.01.patch > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028-ADDENDUM.01.patch, HIVE-22028.01.patch, > HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22028) Clean up Add Partition
[ https://issues.apache.org/jira/browse/HIVE-22028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22028: -- Attachment: (was: HIVE-22028.ADDENDUM.01.patch) > Clean up Add Partition > -- > > Key: HIVE-22028 > URL: https://issues.apache.org/jira/browse/HIVE-22028 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-22028-ADDENDUM.01.patch, HIVE-22028.01.patch, > HIVE-22028.02.patch, HIVE-22028.03.patch, HIVE-22028.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > AlterTableAddPartitionDesc should be immutable, like the rest of the desc > classes. This can not be done 100% right now, as it requires the refactoring > of the ImportSemanticAnalyzer, so the task will be finished only then. > Add Partition logic should be moved from Hive.java to > AlterTableAddPartitionOperation.java, only the metastore calls should remain > in Hive.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098604#comment-17098604
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001956/HIVE-21304.38.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17201 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=99)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=204)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22104/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22104/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22104/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001956 - PreCommit-HIVE-Build
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch,
> HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch,
> HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch,
> HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch,
> HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch,
> HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch,
> HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch,
> HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch,
> HIVE-21304.38.patch, HIVE-21304.38.patch, HIVE-21304.38.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> * Show Bucketing version for ReduceSinkOp in explain extended plan - this
> helps identify what hashing algorithm is being used by by ReduceSinkOp.
> * move the actually selected version to the "conf" so that it doesn't get lost
> * replace trait related logic with a separate optimizer rule
> * do version selection based on a group of operator - this is more reliable
> * skip bucketingversion selection for tables with 1 buckets
> * prefer to use version 2 if possible
> * fix operator creations which didn't set a new conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098599#comment-17098599
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 6 new + 1312 unchanged - 12
fixed = 1318 total (was 1324) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
52s{color} | {color:red} ql generated 1 new + 1530 unchanged - 0 fixed = 1531
total (was 1530) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of table_desc at line 8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:[line 8057] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22104/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22104/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22104/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22104/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098595#comment-17098595
]
Hive QA commented on HIVE-23298:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001955/HIVE-23298.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17200 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer14]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22103/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22103/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22103/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001955 - PreCommit-HIVE-Build
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098589#comment-17098589
]
Hive QA commented on HIVE-23298:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22103/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22103/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23309) Lazy Initialization of Hadoop Shims
[
https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098578#comment-17098578
]
Hive QA commented on HIVE-23309:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001948/HIVE-23309.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17200 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=94)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22102/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22102/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22102/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001948 - PreCommit-HIVE-Build
> Lazy Initialization of Hadoop Shims
> ---
>
> Key: HIVE-23309
> URL: https://issues.apache.org/jira/browse/HIVE-23309
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23309.01.patch, HIVE-23309.02.patch,
> HIVE-23309.03.patch, HIVE-23309.04.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Initialize hadoop-shims only if CM is enabled
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23309) Lazy Initialization of Hadoop Shims
[
https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098571#comment-17098571
]
Hive QA commented on HIVE-23309:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
27s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/server-extensions: The patch generated 0
new + 4 unchanged - 1 fixed = 4 total (was 5) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} hcatalog/server-extensions generated 0 new + 2
unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22102/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C:
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098557#comment-17098557
]
Hive QA commented on HIVE-23340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001947/HIVE-23340.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17200 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22101/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22101/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22101/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001947 - PreCommit-HIVE-Build
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098550#comment-17098550
]
Hive QA commented on HIVE-23340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 4 new + 544 unchanged - 8 fixed = 548 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 186 unchanged - 1 fixed = 186 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22101/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22101/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22101/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.38.patch > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch, > HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch, > HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch, > HIVE-21304.38.patch, HIVE-21304.38.patch, HIVE-21304.38.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098544#comment-17098544
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001945/HIVE-21304.38.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17201 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=94)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
(batchId=222)
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1
(batchId=211)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22100/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22100/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22100/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001945 - PreCommit-HIVE-Build
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch,
> HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch,
> HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch,
> HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch,
> HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch,
> HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch,
> HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch,
> HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch,
> HIVE-21304.38.patch, HIVE-21304.38.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> * Show Bucketing version for ReduceSinkOp in explain extended plan - this
> helps identify what hashing algorithm is being used by by ReduceSinkOp.
> * move the actually selected version to the "conf" so that it doesn't get lost
> * replace trait related logic with a separate optimizer rule
> * do version selection based on a group of operator - this is more reliable
> * skip bucketingversion selection for tables with 1 buckets
> * prefer to use version 2 if possible
> * fix operator creations which didn't set a new conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098541#comment-17098541
]
Jesus Camacho Rodriguez commented on HIVE-23298:
Thanks [~ashutoshc]. Changes LGTM, +1 (pending tests)
Do you have any theory for the number of rows / data size changing (most of the
times to the half)? It should not be related to this patch per se but probably
worth exploring at some point... Maybe it is a side effect of running
annotation of stats before vs after dedup, i.e., two RS after each other is
accounting for double reduction?
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098536#comment-17098536
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 6 new + 1312 unchanged - 12
fixed = 1318 total (was 1324) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
49s{color} | {color:red} ql generated 1 new + 1530 unchanged - 0 fixed = 1531
total (was 1530) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of table_desc at line 8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:[line 8057] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22100/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22100/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22100/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22100/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098522#comment-17098522
]
Hive QA commented on HIVE-23338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001943/HIVE-23338.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17200 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=94)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22099/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22099/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22099/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001943 - PreCommit-HIVE-Build
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Patch Available (was: Open)
[~jcamachorodriguez] changes on https://reviews.apache.org/r/72463/ looks good
to me.
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Open (was: Patch Available)
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Attachment: HIVE-23298.2.patch
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.2.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[
https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098516#comment-17098516
]
Hive QA commented on HIVE-23338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} ptest2 in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
6s{color} | {color:red} ptest2 in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 6s{color}
| {color:red} ptest2 in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
7s{color} | {color:red} ptest2 in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22099/dev-support/hive-personality.sh
|
| git revision | master / b283bbd |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22099/yetus/patch-mvninstall-testutils_ptest2.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22099/yetus/patch-compile-testutils_ptest2.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22099/yetus/patch-compile-testutils_ptest2.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22099/yetus/patch-javadoc-testutils_ptest2.txt
|
| modules | C: standalone-metastore upgrade-acid/pre-upgrade testutils/ptest2
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22099/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bump jackson-databind version up to 2.9.10.4
>
>
> Key: HIVE-23338
> URL: https://issues.apache.org/jira/browse/HIVE-23338
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch
>
>
> com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable.
> And exclude a transitive dependency on
> com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-5134) add tests to partition filter JDO pushdown for like and make sure it works, or remove it
[ https://issues.apache.org/jira/browse/HIVE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5134: --- Fixed via HIVE-22900 > add tests to partition filter JDO pushdown for like and make sure it works, > or remove it > > > Key: HIVE-5134 > URL: https://issues.apache.org/jira/browse/HIVE-5134 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-5134-does-not-work.patch > > > There's a mailing list thread. Partition filtering w/JDO pushdown using LIKE > is not used by Hive due to client check (in PartitionPruner); after enabling > it seems to be broken. We need to fix and enable it, or remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-5134) add tests to partition filter JDO pushdown for like and make sure it works, or remove it
[ https://issues.apache.org/jira/browse/HIVE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-5134. Fix Version/s: 4.0.0 Resolution: Duplicate > add tests to partition filter JDO pushdown for like and make sure it works, > or remove it > > > Key: HIVE-5134 > URL: https://issues.apache.org/jira/browse/HIVE-5134 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-5134-does-not-work.patch > > > There's a mailing list thread. Partition filtering w/JDO pushdown using LIKE > is not used by Hive due to client check (in PartitionPruner); after enabling > it seems to be broken. We need to fix and enable it, or remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23321) Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table
[ https://issues.apache.org/jira/browse/HIVE-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23321: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Tolerate in tests that metastore is not removing rows from the > skewed_string_list_values table > -- > > Key: HIVE-23321 > URL: https://issues.apache.org/jira/browse/HIVE-23321 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23321.01.patch, HIVE-23321.01.patch, > HIVE-23321.01.patch > > > I've found this issue; and filed: HIVE-23289 - however fixing it is not easy > at all.. > * foreign keys were not used > * there were 0 effort to remove these rows before...I don't think when the > feature was added the removal was considered... > * unfortunately designed table hierarchy makes it hard to work with it... > * unclear contracts (at least to me) between skewed_values and sd stuff -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23321) Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table
[ https://issues.apache.org/jira/browse/HIVE-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098505#comment-17098505 ] Ashutosh Chauhan commented on HIVE-23321: - +1 over time we shall remove these arcane and mostly unused features of skewed values and list bucketing HIVE-17852 > Tolerate in tests that metastore is not removing rows from the > skewed_string_list_values table > -- > > Key: HIVE-23321 > URL: https://issues.apache.org/jira/browse/HIVE-23321 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23321.01.patch, HIVE-23321.01.patch, > HIVE-23321.01.patch > > > I've found this issue; and filed: HIVE-23289 - however fixing it is not easy > at all.. > * foreign keys were not used > * there were 0 effort to remove these rows before...I don't think when the > feature was added the removal was considered... > * unfortunately designed table hierarchy makes it hard to work with it... > * unclear contracts (at least to me) between skewed_values and sd stuff -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23344) Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
[
https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098503#comment-17098503
]
Hive QA commented on HIVE-23344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001944/HIVE-23344.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 17200 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestHiveSparkClient.testSetJobGroupAndDescription
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.TestLocalHiveSparkClient.testMultiSessionSparkContextReUse
(batchId=288)
org.apache.hadoop.hive.ql.exec.spark.TestSparkInvalidFileFormat.readTextFileAsParquet
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.TestSparkPlan.testSetRDDCallSite
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testForceConfCloning
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testGetHiveException
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testGetSessionId
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testServerPortAssignment
(batchId=289)
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
(batchId=289)
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testCounters (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testErrorJob (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testErrorJobNotSerializable
(batchId=302)
org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob (batchId=302)
org.apache.hive.spark.client.TestSparkClient.testSyncRpc (batchId=302)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22098/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22098/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22098/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001944 - PreCommit-HIVE-Build
> Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
> -
>
> Key: HIVE-23344
> URL: https://issues.apache.org/jira/browse/HIVE-23344
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch
>
>
> Lots of
> org.scala-lang:scala-compiler:jar:2.11.0:compile ->
> org.scala-lang:scalap:jar:2.11.0:compile ->
> org.json4s:json4s-core_2.11:jar:3.2.11:compile ->
> org.json4s:json4s-jackson_2.11:jar:3.2.11:compile ->
> org.apache.spark:spark-core_2.11:jar:2.3.0:compile
> Where scala-compiler 2.11.0 is a security vulnerability.
> Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2
> dependencies.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23344) Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
[
https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098501#comment-17098501
]
Hive QA commented on HIVE-23344:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22098/dev-support/hive-personality.sh
|
| git revision | master / 69d036f |
| Default Java | 1.8.0_111 |
| modules | C: spark-client ql kryo-registrator . itests/hive-unit
itests/qtest-spark U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22098/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
> -
>
> Key: HIVE-23344
> URL: https://issues.apache.org/jira/browse/HIVE-23344
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch
>
>
> Lots of
> org.scala-lang:scala-compiler:jar:2.11.0:compile ->
> org.scala-lang:scalap:jar:2.11.0:compile ->
> org.json4s:json4s-core_2.11:jar:3.2.11:compile ->
> org.json4s:json4s-jackson_2.11:jar:3.2.11:compile ->
> org.apache.spark:spark-core_2.11:jar:2.3.0:compile
> Where scala-compiler 2.11.0 is a security vulnerability.
> Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2
> dependencies.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23252) Change spark related tests to be optional
[ https://issues.apache.org/jira/browse/HIVE-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23252: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Change spark related tests to be optional > - > > Key: HIVE-23252 > URL: https://issues.apache.org/jira/browse/HIVE-23252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch > > > HIVE-23137 have disabled the execution of some spark related tests; but they > would be still considered by a plain maven command - and the spark artifacts > are (unneccessarily) still downloaded -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23252) Change spark related tests to be optional
[ https://issues.apache.org/jira/browse/HIVE-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098497#comment-17098497 ] Ashutosh Chauhan commented on HIVE-23252: - +1 > Change spark related tests to be optional > - > > Key: HIVE-23252 > URL: https://issues.apache.org/jira/browse/HIVE-23252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch > > > HIVE-23137 have disabled the execution of some spark related tests; but they > would be still considered by a plain maven command - and the spark artifacts > are (unneccessarily) still downloaded -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23284) Remove dependency on mariadb-java-client
[
https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098483#comment-17098483
]
Hive QA commented on HIVE-23284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001942/HIVE-23284.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17198 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22097/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22097/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22097/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001942 - PreCommit-HIVE-Build
> Remove dependency on mariadb-java-client
>
>
> Key: HIVE-23284
> URL: https://issues.apache.org/jira/browse/HIVE-23284
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch,
> HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch,
> HIVE-23284.03.patch, HIVE-23284.03.patch, HIVE-23284.03.patch,
> HIVE-23284.03.patch
>
>
> It has GNU Lesser General Public License which is [Category
> X|https://www.apache.org/legal/resolved.html#category-x].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23284) Remove dependency on mariadb-java-client
[
https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098474#comment-17098474
]
Hive QA commented on HIVE-23284:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22097/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore standalone-metastore/metastore-server U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22097/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove dependency on mariadb-java-client
>
>
> Key: HIVE-23284
> URL: https://issues.apache.org/jira/browse/HIVE-23284
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch,
> HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch,
> HIVE-23284.03.patch, HIVE-23284.03.patch, HIVE-23284.03.patch,
> HIVE-23284.03.patch
>
>
> It has GNU Lesser General Public License which is [Category
> X|https://www.apache.org/legal/resolved.html#category-x].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23355) Fix quoting behavior when masking is enabled
[
https://issues.apache.org/jira/browse/HIVE-23355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23355:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~jfs]!
> Fix quoting behavior when masking is enabled
>
>
> Key: HIVE-23355
> URL: https://issues.apache.org/jira/browse/HIVE-23355
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23355.1.patch
>
>
> Test framework repro:
> {code:java}
> --- a/ql/src/test/queries/clientpositive/masking_reserved.q
> +++ b/ql/src/test/queries/clientpositive/masking_reserved.q
> @@ -5,6 +5,7 @@ set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.autho
> create table keyword_test_off (id int, `etad` string, key int);
> create table keyword_test_on (id int, `date` string, key int);
> create table masking_test_n_masking_reserved (id int, value string, key int);
> +create temporary table masking_test_n_masking_temp stored as orc as select *
> from masking_test_n_masking_reserved; explain select a.`etad`, b.value from
> keyword_test_off a join masking_test_n_masking_reserved b on b.id = a.id;
> select a.`etad`, b.value from keyword_test_off a join
> masking_test_n_masking_reserved b on b.id = a.id; {code}
> fails with:
> {code:java}
> 2020-04-30T20:19:06,833 INFO [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> parse.CalcitePlanner: Creating table default.masking_test_n_masking_temp
> position=23
> 2020-04-30T20:19:06,851 ERROR [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> ql.Driver: FAILED: SemanticException Unrecognized file format in STORED AS
> clause: '`ORC`'
> org.apache.hadoop.hive.ql.parse.SemanticException: Unrecognized file format
> in STORED AS clause: '`ORC`'
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.processStorageFormat(StorageFormat.java:89)
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:74)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13351)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12256)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12471)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:432)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:184) {code}
> We should either not quote the STORED BY argument in this path OR strip
> quotes in processStorageFormat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23355) Fix quoting behavior when masking is enabled
[
https://issues.apache.org/jira/browse/HIVE-23355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098465#comment-17098465
]
Jesus Camacho Rodriguez commented on HIVE-23355:
+1
> Fix quoting behavior when masking is enabled
>
>
> Key: HIVE-23355
> URL: https://issues.apache.org/jira/browse/HIVE-23355
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23355.1.patch
>
>
> Test framework repro:
> {code:java}
> --- a/ql/src/test/queries/clientpositive/masking_reserved.q
> +++ b/ql/src/test/queries/clientpositive/masking_reserved.q
> @@ -5,6 +5,7 @@ set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.autho
> create table keyword_test_off (id int, `etad` string, key int);
> create table keyword_test_on (id int, `date` string, key int);
> create table masking_test_n_masking_reserved (id int, value string, key int);
> +create temporary table masking_test_n_masking_temp stored as orc as select *
> from masking_test_n_masking_reserved; explain select a.`etad`, b.value from
> keyword_test_off a join masking_test_n_masking_reserved b on b.id = a.id;
> select a.`etad`, b.value from keyword_test_off a join
> masking_test_n_masking_reserved b on b.id = a.id; {code}
> fails with:
> {code:java}
> 2020-04-30T20:19:06,833 INFO [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> parse.CalcitePlanner: Creating table default.masking_test_n_masking_temp
> position=23
> 2020-04-30T20:19:06,851 ERROR [bba048b2-380a-46e2-b283-52c1090a0d5c main]
> ql.Driver: FAILED: SemanticException Unrecognized file format in STORED AS
> clause: '`ORC`'
> org.apache.hadoop.hive.ql.parse.SemanticException: Unrecognized file format
> in STORED AS clause: '`ORC`'
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.processStorageFormat(StorageFormat.java:89)
> at
> org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:74)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13351)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12256)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12471)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:432)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:184) {code}
> We should either not quote the STORED BY argument in this path OR strip
> quotes in processStorageFormat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23323) Add qsplits profile
[
https://issues.apache.org/jira/browse/HIVE-23323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098462#comment-17098462
]
Hive QA commented on HIVE-23323:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001929/HIVE-23323.02.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17202 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22096/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22096/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22096/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001929 - PreCommit-HIVE-Build
> Add qsplits profile
> ---
>
> Key: HIVE-23323
> URL: https://issues.apache.org/jira/browse/HIVE-23323
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch,
> HIVE-23323.02.patch, HIVE-23323.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> provide an in-maven way to split-up big testclasses into smaller ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23323) Add qsplits profile
[
https://issues.apache.org/jira/browse/HIVE-23323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098449#comment-17098449
]
Hive QA commented on HIVE-23323:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/util: The patch generated 10 new + 0 unchanged
- 0 fixed = 10 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/qtest: The patch generated 1 new + 8 unchanged
- 0 fixed = 9 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 10 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} itests/util generated 1 new + 53 unchanged - 0 fixed =
54 total (was 53) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:itests/util |
| | Redundant nullcheck of c, which is known to be non-null in
org.apache.hadoop.hive.cli.control.SplitSupport.isSplit0ClassExistsFor(Class)
Redundant null check at SplitSupport.java:is known to be non-null in
org.apache.hadoop.hive.cli.control.SplitSupport.isSplit0ClassExistsFor(Class)
Redundant null check at SplitSupport.java:[line 63] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22096/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22096/yetus/diff-checkstyle-itests_util.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22096/yetus/diff-checkstyle-itests_qtest.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22096/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22096/yetus/new-findbugs-itests_util.html
|
| modules | C: itests itests/util itests/qtest itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22096/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add qsplits profile
> ---
>
> Key: HIVE-23323
>
[jira] [Updated] (HIVE-23309) Lazy Initialization of Hadoop Shims
[ https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23309: --- Status: In Progress (was: Patch Available) > Lazy Initialization of Hadoop Shims > --- > > Key: HIVE-23309 > URL: https://issues.apache.org/jira/browse/HIVE-23309 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23309.01.patch, HIVE-23309.02.patch, > HIVE-23309.03.patch, HIVE-23309.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Initialize hadoop-shims only if CM is enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23309) Lazy Initialization of Hadoop Shims
[ https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23309: --- Attachment: HIVE-23309.04.patch Status: Patch Available (was: In Progress) > Lazy Initialization of Hadoop Shims > --- > > Key: HIVE-23309 > URL: https://issues.apache.org/jira/browse/HIVE-23309 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23309.01.patch, HIVE-23309.02.patch, > HIVE-23309.03.patch, HIVE-23309.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Initialize hadoop-shims only if CM is enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: HIVE-23340.2.patch > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: In Progress (was: Patch Available) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: Patch Available (was: In Progress) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23321) Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table
[
https://issues.apache.org/jira/browse/HIVE-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098435#comment-17098435
]
Hive QA commented on HIVE-23321:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001928/HIVE-23321.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17198 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22095/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22095/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22095/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001928 - PreCommit-HIVE-Build
> Tolerate in tests that metastore is not removing rows from the
> skewed_string_list_values table
> --
>
> Key: HIVE-23321
> URL: https://issues.apache.org/jira/browse/HIVE-23321
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23321.01.patch, HIVE-23321.01.patch,
> HIVE-23321.01.patch
>
>
> I've found this issue; and filed: HIVE-23289 - however fixing it is not easy
> at all..
> * foreign keys were not used
> * there were 0 effort to remove these rows before...I don't think when the
> feature was added the removal was considered...
> * unfortunately designed table hierarchy makes it hard to work with it...
> * unclear contracts (at least to me) between skewed_values and sd stuff
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23321) Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table
[
https://issues.apache.org/jira/browse/HIVE-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098419#comment-17098419
]
Hive QA commented on HIVE-23321:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22095/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22095/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tolerate in tests that metastore is not removing rows from the
> skewed_string_list_values table
> --
>
> Key: HIVE-23321
> URL: https://issues.apache.org/jira/browse/HIVE-23321
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23321.01.patch, HIVE-23321.01.patch,
> HIVE-23321.01.patch
>
>
> I've found this issue; and filed: HIVE-23289 - however fixing it is not easy
> at all..
> * foreign keys were not used
> * there were 0 effort to remove these rows before...I don't think when the
> feature was added the removal was considered...
> * unfortunately designed table hierarchy makes it hard to work with it...
> * unclear contracts (at least to me) between skewed_values and sd stuff
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23252) Change spark related tests to be optional
[
https://issues.apache.org/jira/browse/HIVE-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098415#comment-17098415
]
Hive QA commented on HIVE-23252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001930/HIVE-23252.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17198 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22094/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22094/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22094/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001930 - PreCommit-HIVE-Build
> Change spark related tests to be optional
> -
>
> Key: HIVE-23252
> URL: https://issues.apache.org/jira/browse/HIVE-23252
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch
>
>
> HIVE-23137 have disabled the execution of some spark related tests; but they
> would be still considered by a plain maven command - and the spark artifacts
> are (unneccessarily) still downloaded
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23252) Change spark related tests to be optional
[
https://issues.apache.org/jira/browse/HIVE-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098413#comment-17098413
]
Hive QA commented on HIVE-23252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22094/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| modules | C: . itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22094/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change spark related tests to be optional
> -
>
> Key: HIVE-23252
> URL: https://issues.apache.org/jira/browse/HIVE-23252
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch,
> HIVE-23252.01.patch
>
>
> HIVE-23137 have disabled the execution of some spark related tests; but they
> would be still considered by a plain maven command - and the spark artifacts
> are (unneccessarily) still downloaded
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23352) More user friendly StorageAuthorizationProvider log messages
[
https://issues.apache.org/jira/browse/HIVE-23352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098411#comment-17098411
]
Panagiotis Garefalakis commented on HIVE-23352:
---
Hey [~thejas] – the behaviour is indeed correct (i.e., if we would really want
to disable authorization for external tables we could have set
_hive.metastore.authorization.storage.check.externaltable.drop_ to *false*)
However, I find the messages a bit confusing.
As a first step, we could make messages distinguish permissions over dirs/paths
versus tables – as in when dropping an external table we are not really
deleting a path (see LOG message above), only the metadata, so the user should
know that its a table permission issue and not a dir permission issue.
> More user friendly StorageAuthorizationProvider log messages
>
>
> Key: HIVE-23352
> URL: https://issues.apache.org/jira/browse/HIVE-23352
> Project: Hive
> Issue Type: Improvement
> Components: Security
>Affects Versions: 4.0.0
>Reporter: Panagiotis Garefalakis
>Priority: Minor
>
> Currently *StorageBasedAuthorizationProvider* returns messages (like below)
> about data paths even for _External_ tables where a drop command would just
> remove metadata. Lets make those messages more user-friendly.
> {code:java}
> Permission Denied: User hive can't delete hdfs://XXX.com:8020/tmp/testuser
> because sticky bit is set on the parent dir and user does not own this file
> or its parent)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.38.patch > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch, > HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch, > HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch, > HIVE-21304.38.patch, HIVE-21304.38.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098381#comment-17098381
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001926/HIVE-21304.38.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17199 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22093/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22093/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22093/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001926 - PreCommit-HIVE-Build
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch,
> HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch,
> HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch,
> HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch,
> HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch,
> HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch,
> HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch,
> HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch,
> HIVE-21304.38.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> * Show Bucketing version for ReduceSinkOp in explain extended plan - this
> helps identify what hashing algorithm is being used by by ReduceSinkOp.
> * move the actually selected version to the "conf" so that it doesn't get lost
> * replace trait related logic with a separate optimizer rule
> * do version selection based on a group of operator - this is more reliable
> * skip bucketingversion selection for tables with 1 buckets
> * prefer to use version 2 if possible
> * fix operator creations which didn't set a new conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23338: - Attachment: HIVE-23338.01.patch Status: Patch Available (was: Open) > Bump jackson-databind version up to 2.9.10.4 > > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23338.01.patch, HIVE-23338.01.patch > > > com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable. > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Status: Open (was: Patch Available) > Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency > - > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch > > > Lots of > org.scala-lang:scala-compiler:jar:2.11.0:compile -> > org.scala-lang:scalap:jar:2.11.0:compile -> > org.json4s:json4s-core_2.11:jar:3.2.11:compile -> > org.json4s:json4s-jackson_2.11:jar:3.2.11:compile -> > org.apache.spark:spark-core_2.11:jar:2.3.0:compile > Where scala-compiler 2.11.0 is a security vulnerability. > Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2 > dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Attachment: HIVE-23344.01.patch Status: Patch Available (was: Open) > Remove org.scala-lang:scala-compiler 2.11.0 transitive dependency > - > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch > > > Lots of > org.scala-lang:scala-compiler:jar:2.11.0:compile -> > org.scala-lang:scalap:jar:2.11.0:compile -> > org.json4s:json4s-core_2.11:jar:3.2.11:compile -> > org.json4s:json4s-jackson_2.11:jar:3.2.11:compile -> > org.apache.spark:spark-core_2.11:jar:2.3.0:compile > Where scala-compiler 2.11.0 is a security vulnerability. > Excluding org.json4s:json4s-jackson_2.11:jar:3.2.11 from spark-core_2 > dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson-databind version up to 2.9.10.4
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23338: - Status: Open (was: Patch Available) > Bump jackson-databind version up to 2.9.10.4 > > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23338.01.patch > > > com.fasterxml.jackson.core:jackson-databind:2.9.9 is exploitable. > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5, which is also exploitable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23284) Remove dependency on mariadb-java-client
[ https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23284: - Status: Open (was: Patch Available) > Remove dependency on mariadb-java-client > > > Key: HIVE-23284 > URL: https://issues.apache.org/jira/browse/HIVE-23284 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch, > HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch, > HIVE-23284.03.patch, HIVE-23284.03.patch, HIVE-23284.03.patch, > HIVE-23284.03.patch > > > It has GNU Lesser General Public License which is [Category > X|https://www.apache.org/legal/resolved.html#category-x]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23284) Remove dependency on mariadb-java-client
[ https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23284: - Attachment: HIVE-23284.03.patch Status: Patch Available (was: Open) > Remove dependency on mariadb-java-client > > > Key: HIVE-23284 > URL: https://issues.apache.org/jira/browse/HIVE-23284 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch, > HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch, > HIVE-23284.03.patch, HIVE-23284.03.patch, HIVE-23284.03.patch, > HIVE-23284.03.patch > > > It has GNU Lesser General Public License which is [Category > X|https://www.apache.org/legal/resolved.html#category-x]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-5134) add tests to partition filter JDO pushdown for like and make sure it works, or remove it
[ https://issues.apache.org/jira/browse/HIVE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098366#comment-17098366 ] Syed Shameerur Rahman commented on HIVE-5134: - [~bargitta] Yes, Refer: https://issues.apache.org/jira/browse/HIVE-22900 > add tests to partition filter JDO pushdown for like and make sure it works, > or remove it > > > Key: HIVE-5134 > URL: https://issues.apache.org/jira/browse/HIVE-5134 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-5134-does-not-work.patch > > > There's a mailing list thread. Partition filtering w/JDO pushdown using LIKE > is not used by Hive due to client check (in PartitionPruner); after enabling > it seems to be broken. We need to fix and enable it, or remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-5134) add tests to partition filter JDO pushdown for like and make sure it works, or remove it
[ https://issues.apache.org/jira/browse/HIVE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098365#comment-17098365 ] Xi Chen commented on HIVE-5134: --- does like operator works? > add tests to partition filter JDO pushdown for like and make sure it works, > or remove it > > > Key: HIVE-5134 > URL: https://issues.apache.org/jira/browse/HIVE-5134 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-5134-does-not-work.patch > > > There's a mailing list thread. Partition filtering w/JDO pushdown using LIKE > is not used by Hive due to client check (in PartitionPruner); after enabling > it seems to be broken. We need to fix and enable it, or remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098360#comment-17098360
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
30s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 6 new + 1312 unchanged - 12
fixed = 1318 total (was 1324) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
44s{color} | {color:red} ql generated 1 new + 1530 unchanged - 0 fixed = 1531
total (was 1530) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of table_desc at line 8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:8064 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:[line 8057] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22093/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22093/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22093/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22093/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098350#comment-17098350
]
Hive QA commented on HIVE-23298:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001925/HIVE-23298.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 17198 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join18]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer14]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[distinct_groupby]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[distinct_stats]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[except_all]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby3_map]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby3_map_multi_distinct]
(batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby3_map_skew]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby4_map]
(batchId=88)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby4_map_skew]
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby5_map]
(batchId=93)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby5_map_skew]
(batchId=60)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_multiinsert]
(batchId=77)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22092/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22092/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22092/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001925 - PreCommit-HIVE-Build
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098318#comment-17098318
]
Hive QA commented on HIVE-23298:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
31s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22092/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22092/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098309#comment-17098309
]
Hive QA commented on HIVE-23347:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001919/HIVE-23347.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17199 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=48)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[7]
(batchId=141)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22091/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22091/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22091/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001919 - PreCommit-HIVE-Build
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1; --> Returns zero partitions
> 5. select * from t1; --> Returns empty data.
> When the partition directory names are changed to lower case, this works fine.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098306#comment-17098306
]
Hive QA commented on HIVE-23347:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
35s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22091/dev-support/hive-personality.sh
|
| git revision | master / 1f4bc02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22091/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external table t1 (key int, value string)
[jira] [Updated] (HIVE-23252) Change spark related tests to be optional
[ https://issues.apache.org/jira/browse/HIVE-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23252: Attachment: HIVE-23252.01.patch > Change spark related tests to be optional > - > > Key: HIVE-23252 > URL: https://issues.apache.org/jira/browse/HIVE-23252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch, HIVE-23252.01.patch, HIVE-23252.01.patch, > HIVE-23252.01.patch > > > HIVE-23137 have disabled the execution of some spark related tests; but they > would be still considered by a plain maven command - and the spark artifacts > are (unneccessarily) still downloaded -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23321) Tolerate in tests that metastore is not removing rows from the skewed_string_list_values table
[ https://issues.apache.org/jira/browse/HIVE-23321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23321: Attachment: HIVE-23321.01.patch > Tolerate in tests that metastore is not removing rows from the > skewed_string_list_values table > -- > > Key: HIVE-23321 > URL: https://issues.apache.org/jira/browse/HIVE-23321 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23321.01.patch, HIVE-23321.01.patch, > HIVE-23321.01.patch > > > I've found this issue; and filed: HIVE-23289 - however fixing it is not easy > at all.. > * foreign keys were not used > * there were 0 effort to remove these rows before...I don't think when the > feature was added the removal was considered... > * unfortunately designed table hierarchy makes it hard to work with it... > * unclear contracts (at least to me) between skewed_values and sd stuff -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23323) Add qsplits profile
[ https://issues.apache.org/jira/browse/HIVE-23323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23323: Attachment: HIVE-23323.02.patch > Add qsplits profile > --- > > Key: HIVE-23323 > URL: https://issues.apache.org/jira/browse/HIVE-23323 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23323.01.patch, HIVE-23323.02.patch, > HIVE-23323.02.patch, HIVE-23323.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > > provide an in-maven way to split-up big testclasses into smaller ones -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23357) Allow queue user to be configured separately from doAs
[
https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098291#comment-17098291
]
Hive QA commented on HIVE-23357:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001918/HIVE-23357.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17199 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22090/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22090/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22090/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001918 - PreCommit-HIVE-Build
> Allow queue user to be configured separately from doAs
> --
>
> Key: HIVE-23357
> URL: https://issues.apache.org/jira/browse/HIVE-23357
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23357.1.patch
>
>
> There are cases where we may want to submit to a YARN queue as the logged in
> user but not require full doas for writing of HDFS or authorization.
> A setting could be added to control this behavior:
> set hive.server2.tez.queue.access.method=default
> default is to follow doas setting (the current method)
> loggedin is to always username of logged in user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23215) Make FilterContext and MutableFilterContext interfaces
[ https://issues.apache.org/jira/browse/HIVE-23215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23215: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Owen & Panos! > Make FilterContext and MutableFilterContext interfaces > -- > > Key: HIVE-23215 > URL: https://issues.apache.org/jira/browse/HIVE-23215 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23215.01.patch, HIVE-23215.02.patch, > HIVE-23215.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > HIVE-22959 introduced FilterContext to support ORC-577. The duplication of > fields between the FilterContext and VectorizedRowBatch seems likely to cause > user confusion. This patch makes them interfaces that VectorizedRowBatch > implements. > Thus, there is a single copy of the data and no need to copy them back and > forth. LLAP can make its own implementation of the interfaces if it doesn't > want to use VectorizedRowBatch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23322) Update some tests to better tolerate a more busy environment
[ https://issues.apache.org/jira/browse/HIVE-23322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23322: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Update some tests to better tolerate a more busy environment > > > Key: HIVE-23322 > URL: https://issues.apache.org/jira/browse/HIVE-23322 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23322.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23322) Update some tests to better tolerate a more busy environment
[ https://issues.apache.org/jira/browse/HIVE-23322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098287#comment-17098287 ] Ashutosh Chauhan commented on HIVE-23322: - +1 > Update some tests to better tolerate a more busy environment > > > Key: HIVE-23322 > URL: https://issues.apache.org/jira/browse/HIVE-23322 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23322.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.38.patch > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch, HIVE-21304.33.patch, > HIVE-21304.34.patch, HIVE-21304.34.patch, HIVE-21304.35.patch, > HIVE-21304.35.patch, HIVE-21304.36.patch, HIVE-21304.37.patch, > HIVE-21304.38.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23240) loadDynamicPartition complains about static partitions even when they are provided in the description
[ https://issues.apache.org/jira/browse/HIVE-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23240: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. > loadDynamicPartition complains about static partitions even when they are > provided in the description > -- > > Key: HIVE-23240 > URL: https://issues.apache.org/jira/browse/HIVE-23240 > Project: Hive > Issue Type: Bug >Reporter: Reza Safi >Assignee: Reza Safi >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-23240.2.patch, HIVE-23240.3.patch, > HIVE-23240.4.patch, HIVE-23240.patch > > > Hive is computing valid dynamic partitions here: > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2853 > However it later uses the specification provided by client here: > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2879 > (partSpec is exactly what client has provided and partSpec.keySet() contains > both static and dynamic partitions key) > As a result the makeSpecFromName here will expect both static and dynamic > partitions in requiredKeys: > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java#L580 > However since the curPath that is passed to the above method is just like > "somePath/dynamicPart=value" which miss the static partitions and a result > the method will ignore static partition keys then complains in log a warning > that the static partition keys are missing. Returning false to Hive.java, a > log warning that "dynamicPart=value" is an invalid partition will be issued, > despite the fact that the dynamic partition has been validated before: > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2880 > > This will cause a silent data corruption in some clients. As an example spark > will suffer from this when working with hive metastore in master branch. > It seems that if the goal was just to warn the client, there is no need to > ignore the valid dynamic partition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22737:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.5.patch,
> HIVE-22737.6.patch, HIVE-22737.7.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Patch Available (was: Open)
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Attachment: HIVE-23298.1.patch
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.1.patch, HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23298) Disable RS deduplication step in Optimizer if it is run in TezCompiler
[
https://issues.apache.org/jira/browse/HIVE-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23298:
Status: Open (was: Patch Available)
> Disable RS deduplication step in Optimizer if it is run in TezCompiler
> --
>
> Key: HIVE-23298
> URL: https://issues.apache.org/jira/browse/HIVE-23298
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23298.patch
>
>
> HIVE-20703 introduced an additional RS deduplication step in TezCompiler. We
> could possibly try to disable the one that runs in {{Optimizer}} if we are
> using Tez so we do not run the optimization twice.
> This issue is to explore that possibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23357) Allow queue user to be configured separately from doAs
[
https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098276#comment-17098276
]
Hive QA commented on HIVE-23357:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 8 new + 10 unchanged - 0 fixed
= 18 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22090/dev-support/hive-personality.sh
|
| git revision | master / 659e28d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22090/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22090/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow queue user to be configured separately from doAs
> --
>
> Key: HIVE-23357
> URL: https://issues.apache.org/jira/browse/HIVE-23357
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23357.1.patch
>
>
> There are cases where we may want to submit to a YARN queue as the logged in
> user but not require full doas for writing of HDFS or authorization.
> A setting could be added to control this behavior:
> set hive.server2.tez.queue.access.method=default
> default is to follow doas setting (the current method)
> loggedin is to always username of logged in user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098267#comment-17098267
]
Hive QA commented on HIVE-22737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001917/HIVE-22737.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17199 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22089/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22089/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22089/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001917 - PreCommit-HIVE-Build
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.5.patch,
> HIVE-22737.6.patch, HIVE-22737.7.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098256#comment-17098256
]
Hive QA commented on HIVE-22737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 19 new + 27 unchanged - 0
fixed = 46 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22089/dev-support/hive-personality.sh
|
| git revision | master / 659e28d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22089/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22089/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.5.patch,
> HIVE-22737.6.patch, HIVE-22737.7.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
