[jira] [Commented] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[
https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978169#comment-16978169
]
Hive QA commented on HIVE-22505:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 6 new + 397 unchanged - 0
fixed = 403 total (was 397) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19507/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19507/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19507/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ClassCastException caused by wrong Vectorized operator selection
>
>
> Key: HIVE-22505
> URL: https://issues.apache.org/jira/browse/HIVE-22505
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch,
> HIVE-22505.4.patch, HIVE-22505.patch, query_error.out,
> query_vector_explain.out, vectorized_join.q
>
>
> VectorMapJoinOuterFilteredOperator does not currently support full outer
> joins but using the current Vectorizer logic it can be selected when a there
> is a filter involved. This can make queries fail with ClassCastException when
> their data and metadata in the VectorMapJoinOuterFilteredOperator do not
> match.
> The query attached demonstrates the issue and the log attached shows the
> java.lang.ClassCastException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[
https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978149#comment-16978149
]
Hive QA commented on HIVE-22510:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986277/HIVE-22510.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17711 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=64)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=140)
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986277 - PreCommit-HIVE-Build
> Support decimal64 operations for column operands with different scales
> --
>
> Key: HIVE-22510
> URL: https://issues.apache.org/jira/browse/HIVE-22510
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22510.2.patch, HIVE-22510.3.patch
>
>
> Right now, if the operands on the decimal64 operations are columns with
> different scales, then we do not use the decimal64 vectorized version and
> fall back to HiveDecimal vectorized version of the operator. In this Jira, we
> will check if we can use decimal64 vectorized version, even if the scales are
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: HIVE-22505.4.patch > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.4.patch, HIVE-22505.patch, query_error.out, > query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Patch Available (was: Open) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.4.patch, HIVE-22505.patch, query_error.out, > query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Open (was: Patch Available) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.4.patch, HIVE-22505.patch, query_error.out, > query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Patch Available (was: Open) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Open (was: Patch Available) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: HIVE-22505.4.patch.txt > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: (was: HIVE-22505.4.patch.txt) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Patch Available (was: Open) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Open (was: Patch Available) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Open (was: Patch Available) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Patch Available (was: Open) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22317) Beeline-site parser does not handle the variable substitution correctly
[
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978109#comment-16978109
]
mahesh kumar behera edited comment on HIVE-22317 at 11/20/19 6:44 AM:
--
[~Rajkumar Singh]
Code changes looks fine to me. Add a comment saying it is required for the
variable config replacement case, as its not very intuitive to use conf.key
when key.value is already present. Please add a test case also for the same.
was (Author: maheshk114):
[~Rajkumar Singh]
Code changes looks fine to me. Please add a test case also for the same.
> Beeline-site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>
>
> beeline.hs2.jdbc.url.container
>
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>
>
>
> beeline.hs2.jdbc.url.default
> test
>
>
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>
>
> beeline.hs2.jdbc.url.llap
>
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>
>
> {code}
> beeline fail to connect because it does not parse the substituted value
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url:
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22317) Beeline-site parser does not handle the variable substitution correctly
[
https://issues.apache.org/jira/browse/HIVE-22317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978109#comment-16978109
]
mahesh kumar behera commented on HIVE-22317:
[~Rajkumar Singh]
Code changes looks fine to me. Please add a test case also for the same.
> Beeline-site parser does not handle the variable substitution correctly
> ---
>
> Key: HIVE-22317
> URL: https://issues.apache.org/jira/browse/HIVE-22317
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 4.0.0
> Environment: Hive-4.0.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22317.patch
>
>
> beeline-site.xml
> {code:java}
> http://www.w3.org/2001/XInclude;>
>
>
> beeline.hs2.jdbc.url.container
>
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>
>
>
> beeline.hs2.jdbc.url.default
> test
>
>
> beeline.hs2.jdbc.url.test
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue
>
>
> beeline.hs2.jdbc.url.llap
>
> jdbc:hive2://c3220-node2.host.com:2181,c3220-node3.host.com:2181,c3220-node4.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive
>
>
> {code}
> beeline fail to connect because it does not parse the substituted value
> correctly
> {code:java}
> beeline
> Error in parsing jdbc url:
> ${beeline.hs2.jdbc.url.container}?tez.queue.name=myqueue from beeline-site.xml
> beeline> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22428) Remove superfluous "Failed to get database" WARN Logging in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-22428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978105#comment-16978105
]
Ashutosh Bapat commented on HIVE-22428:
---
[~belugabehr], [~mgergely],
In this commit, you have modified debugLog() and pass (new Exception()) to
LOG.debug() instead of earlier getStackTrace(). (new Exception()) prints the
whole stack trace which can be misled as some error condition/exception. But
getStackTrace() printed just a few frames rationalising the output, which was
easy not to mistake for an actual exception.
I was looking at the debug output of one of my tests and had this confusion. It
took me a bit of time to know that it wasn't a real exception. Here's how it
looked. I thought it was NullPointer exception or something like that.
{code:java}
2019-11-19T08:13:31,392 DEBUG [PrivilegeSynchronizer] metastore.ObjectStore:
Commit transaction: count = 1, isactive true
java.lang.Exception: null
at
org.apache.hadoop.hive.metastore.ObjectStore.debugLog(ObjectStore.java:9671)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:475)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.commit(ObjectStore.java:3707)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3608)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableAllColumnGrants(ObjectStore.java:6521)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6455)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy40.refreshPrivileges(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:7136)
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_222]{code}
Is it possible to revert back to the old method or at least sanitize the output
so that it doesn't look like a real exception?
> Remove superfluous "Failed to get database" WARN Logging in ObjectStore
> ---
>
> Key: HIVE-22428
> URL: https://issues.apache.org/jira/browse/HIVE-22428
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22428.1.patch
>
>
> In my testing, I get lots of logs like this:
> {code:none}
> Line 26319: 2019-10-28T21:09:52,134 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26327: 2019-10-28T21:09:52,135 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26504: 2019-10-28T21:09:52,600 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26519: 2019-10-28T21:09:52,606 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26695: 2019-10-28T21:09:52,922 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26703: 2019-10-28T21:09:52,923 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26763: 2019-10-28T21:09:52,936 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26778: 2019-10-28T21:09:52,939 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26963: 2019-10-28T21:09:53,273 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 26978: 2019-10-28T21:09:53,276 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[
https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978102#comment-16978102
]
Hive QA commented on HIVE-22510:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 2 new + 794 unchanged - 1
fixed = 796 total (was 795) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19506/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19506/yetus/diff-checkstyle-ql.txt
|
| modules | C: vector-code-gen ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19506/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support decimal64 operations for column operands with different scales
> --
>
> Key: HIVE-22510
> URL: https://issues.apache.org/jira/browse/HIVE-22510
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22510.2.patch, HIVE-22510.3.patch
>
>
> Right now, if the operands on the decimal64 operations are columns with
> different scales, then we do not use the decimal64 vectorized version and
> fall back to HiveDecimal vectorized version of the operator. In this Jira, we
> will check if we can use decimal64 vectorized version, even if the scales are
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[
https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978085#comment-16978085
]
Hive QA commented on HIVE-22505:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986270/HIVE-22505.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17709 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=300)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19505/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19505/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19505/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986270 - PreCommit-HIVE-Build
> ClassCastException caused by wrong Vectorized operator selection
>
>
> Key: HIVE-22505
> URL: https://issues.apache.org/jira/browse/HIVE-22505
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch,
> HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q
>
>
> VectorMapJoinOuterFilteredOperator does not currently support full outer
> joins but using the current Vectorizer logic it can be selected when a there
> is a filter involved. This can make queries fail with ClassCastException when
> their data and metadata in the VectorMapJoinOuterFilteredOperator do not
> match.
> The query attached demonstrates the issue and the log attached shows the
> java.lang.ClassCastException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[
https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978062#comment-16978062
]
Hive QA commented on HIVE-22505:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 6 new + 397 unchanged - 0
fixed = 403 total (was 397) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19505/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19505/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19505/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ClassCastException caused by wrong Vectorized operator selection
>
>
> Key: HIVE-22505
> URL: https://issues.apache.org/jira/browse/HIVE-22505
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch,
> HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q
>
>
> VectorMapJoinOuterFilteredOperator does not currently support full outer
> joins but using the current Vectorizer logic it can be selected when a there
> is a filter involved. This can make queries fail with ClassCastException when
> their data and metadata in the VectorMapJoinOuterFilteredOperator do not
> match.
> The query attached demonstrates the issue and the log attached shows the
> java.lang.ClassCastException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20864) No running LLAP daemons! Please check LLAP service status and zookeeper configuration
[
https://issues.apache.org/jira/browse/HIVE-20864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978054#comment-16978054
]
lv haiyang commented on HIVE-20864:
---
[https://github.com/hortonworks/hive-testbench/blob/hdp3/runSuite.pl#L36]
if use llap，the perl will modify,will you?
> No running LLAP daemons! Please check LLAP service status and zookeeper
> configuration
> -
>
> Key: HIVE-20864
> URL: https://issues.apache.org/jira/browse/HIVE-20864
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.0
>Reporter: zhangbutao
>Priority: Major
>
> I test hive3.1.0 using TPCDS. Jobs would be falied after executing some sql,
> and i have checked that LLAP daemons were still alive as well as LLAP zknodes
> still existed in zookeeper. The error log as follows:
> {code:java}
> ]Vertex failed, vertexName=Map 6, vertexId=vertex_1541034860050_0181_194_04,
> diagnostics=[Vertex vertex_1541034860050_0181_194_04 [Map 6] killed/failed
> due to:INIT_FAILURE, Fail to create InputInitializerManager,
> org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class
> with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
> at
> org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4101)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:205)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2912)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2859)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2841)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1939)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:204)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)
> at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
> at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor45.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
> ... 25 more
> Caused by: java.lang.IllegalArgumentException: No running LLAP daemons!
> Please check LLAP service status and zookeeper configuration
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:57)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> ... 29 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978053#comment-16978053
]
Hive QA commented on HIVE-22476:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986265/HIVE-22476.7.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17693 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=112)
[explainanalyze_3.q,orc_vectorization_ppd.q,topnkey.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,hybridgrace_hashjoin_1.q,tez_union_udtf.q,delete_orig_table.q,tez_complextype_with_null.q,hybridgrace_hashjoin_2.q,tez-tag.q,vector_topnkey.q,explainanalyze_5.q,vector_join_part_col_char.q]
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
(batchId=354)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19504/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19504/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986265 - PreCommit-HIVE-Build
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978036#comment-16978036
]
Hive QA commented on HIVE-22476:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19504/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19504/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values
[jira] [Comment Edited] (HIVE-22472) Unable to create hive view
[ https://issues.apache.org/jira/browse/HIVE-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978035#comment-16978035 ] Eric Lin edited comment on HIVE-22472 at 11/20/19 3:26 AM: --- Seems to relate to HIVE-14719, or maybe a duplicate. was (Author: ericlin): Seems to relate to HIVE-14719 > Unable to create hive view > --- > > Key: HIVE-22472 > URL: https://issues.apache.org/jira/browse/HIVE-22472 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.5 >Reporter: Ashok >Priority: Major > > unable to create hive view with EXISTS clause: > Error: > FAILED: SemanticException Line 0:-1 Invalid table alias or column reference > 'sq_1': (possible column names are: _table_or_col lf) file_date) sq_corr_, (. > (tok_table_or_col sq_1) sq_corr_1)) > > Below reproduction steps: > -- Setup Tables > create table bug_part_1 (table_name string, partition_date date, file_date > timestamp); > create table bug_part_2 (id string, file_date timestamp) partitioned by > (partition_date date); > -- Example 1 - Works if just query. > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- Example 2 - Fails in view. > create or replace view bug_view > as > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22472) Unable to create hive view
[ https://issues.apache.org/jira/browse/HIVE-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978035#comment-16978035 ] Eric Lin commented on HIVE-22472: - Seems to relate to HIVE-14719 > Unable to create hive view > --- > > Key: HIVE-22472 > URL: https://issues.apache.org/jira/browse/HIVE-22472 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.5 >Reporter: Ashok >Priority: Major > > unable to create hive view with EXISTS clause: > Error: > FAILED: SemanticException Line 0:-1 Invalid table alias or column reference > 'sq_1': (possible column names are: _table_or_col lf) file_date) sq_corr_, (. > (tok_table_or_col sq_1) sq_corr_1)) > > Below reproduction steps: > -- Setup Tables > create table bug_part_1 (table_name string, partition_date date, file_date > timestamp); > create table bug_part_2 (id string, file_date timestamp) partitioned by > (partition_date date); > -- Example 1 - Works if just query. > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- Example 2 - Fails in view. > create or replace view bug_view > as > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22472) Unable to create hive view
[ https://issues.apache.org/jira/browse/HIVE-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978034#comment-16978034 ] Eric Lin commented on HIVE-22472: - Setting hive.cbo.enable=true can help to avoid the issue, but not sure why yet. > Unable to create hive view > --- > > Key: HIVE-22472 > URL: https://issues.apache.org/jira/browse/HIVE-22472 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.5 >Reporter: Ashok >Priority: Major > > unable to create hive view with EXISTS clause: > Error: > FAILED: SemanticException Line 0:-1 Invalid table alias or column reference > 'sq_1': (possible column names are: _table_or_col lf) file_date) sq_corr_, (. > (tok_table_or_col sq_1) sq_corr_1)) > > Below reproduction steps: > -- Setup Tables > create table bug_part_1 (table_name string, partition_date date, file_date > timestamp); > create table bug_part_2 (id string, file_date timestamp) partitioned by > (partition_date date); > -- Example 1 - Works if just query. > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- Example 2 - Fails in view. > create or replace view bug_view > as > select vlf.id > from bug_part_2 vlf > where 1=1 > and exists ( > select null > from ( > select max(file_date) file_date, max(partition_date) as partition_date > from bug_part_1 > ) lf > where lf.partition_date = vlf.partition_date and lf.file_date = vlf.file_date > ); > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978025#comment-16978025
]
Hive QA commented on HIVE-22435:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986257/HIVE-22435.7.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17710 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19503/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19503/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19503/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986257 - PreCommit-HIVE-Build
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch, HIVE-22435.7.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
> [0:key._col0, 1:key._col1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:538)
> at
>
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977985#comment-16977985
]
Hive QA commented on HIVE-22435:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} ql: The patch generated 0 new + 437 unchanged - 2
fixed = 437 total (was 439) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19503/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19503/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch, HIVE-22435.7.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by
[jira] [Assigned] (HIVE-22515) Support cast to decimal64 in Vectorization
[ https://issues.apache.org/jira/browse/HIVE-22515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-22515: --- Assignee: Ramesh Kumar Thangarajan > Support cast to decimal64 in Vectorization > -- > > Key: HIVE-22515 > URL: https://issues.apache.org/jira/browse/HIVE-22515 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Support cast to decimal64 in Vectorization -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.9.1
[
https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977968#comment-16977968
]
Hive QA commented on HIVE-21737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986256/0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19502/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19502/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19502/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12986256/0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986256 - PreCommit-HIVE-Build
> Upgrade Avro to version 1.9.1
> -
>
> Key: HIVE-21737
> URL: https://issues.apache.org/jira/browse/HIVE-21737
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ismaël Mejía
>Assignee: Fokko Driesprong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch,
> 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner
> version of Avro without Jackson in the public API. Worth the update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22512) Use direct SQL to fetch column privileges in refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-22512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977966#comment-16977966
]
Hive QA commented on HIVE-22512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986254/HIVE-22512.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17707 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19501/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19501/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19501/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986254 - PreCommit-HIVE-Build
> Use direct SQL to fetch column privileges in refreshPrivileges
> --
>
> Key: HIVE-22512
> URL: https://issues.apache.org/jira/browse/HIVE-22512
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Attachments: HIVE-22512.01.patch
>
>
> refreshPrivileges() calls listTableAllColumnGrants() to fetch the column
> level privileges. The later function retrieves the individual column objects
> by firing one query per column privilege object, thus causing the backend db
> to be swamped by these queries when PrivilegeSynchronizer is run.
> PrivilegeSynchronizer synchronizes privileges of all the databases, tables
> and columns and thus the backend db can get swamped really bad when there are
> thousands of tables with hundreds of columns.
> The output of listTableAllColumnGrants() is not used completely so all the
> columns the PM has tried to retrieves anyway goes waste.
> Fix this by using direct SQL to fetch column privileges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-9223) HiveServer2 on Tez doesn't support concurrent queries within one session
[
https://issues.apache.org/jira/browse/HIVE-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976833#comment-16976833
]
Tak-Lon (Stephen) Wu edited comment on HIVE-9223 at 11/20/19 12:54 AM:
---
not sure if this is the same scope of this JIRA, but
[HUE-9083|https://issues.cloudera.org/browse/HUE-9083] has fixed the
{{max_number_of_sessions > 1}} issue such user does not submit multiple query
to the same Tez session. (will that help to resolve this issue ?)
was (Author: taklwu):
not sure if this is the same scope of this JIRA, but
[HUE-9083|https://issues.cloudera.org/browse/HUE-9083] has fixed the
{{max_number_of_sessions > 1}} issue such user does not submit multiple query
to the same Tez session. (will that remove this issue ?)
> HiveServer2 on Tez doesn't support concurrent queries within one session
>
>
> Key: HIVE-9223
> URL: https://issues.apache.org/jira/browse/HIVE-9223
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Pala M Muthaia
>Priority: Major
>
> When a user submits multiple queries in the same HS2 session (using thrift
> interface) concurrently, the query goes through the same TezSessionState and
> ends up being submitted to the same Tez AM, and the second query fails with
> the error "App master already running a DAG"
> Is this by design? I looked into the code, and the comments as well as the
> code suggest support only for serial execution of queries within the same
> HiveServer2 session (on tez).
> This works for CLI environment but in a server, it is plausible that client
> sends multiple concurrent queries under the same session (e.g: a web app that
> executes queries for user, such as Cloudera Hue). So shouldn't HS2 on Tez
> implementation support concurrent queries?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22512) Use direct SQL to fetch column privileges in refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-22512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977940#comment-16977940
]
Hive QA commented on HIVE-22512:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
22s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 19 new + 615 unchanged - 6 fixed = 634 total (was 621) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19501/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19501/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19501/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use direct SQL to fetch column privileges in refreshPrivileges
> --
>
> Key: HIVE-22512
> URL: https://issues.apache.org/jira/browse/HIVE-22512
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Attachments: HIVE-22512.01.patch
>
>
> refreshPrivileges() calls listTableAllColumnGrants() to fetch the column
> level privileges. The later function retrieves the individual column objects
> by firing one query per column privilege object, thus causing the backend db
> to be swamped by these queries when PrivilegeSynchronizer is run.
> PrivilegeSynchronizer synchronizes privileges of all the databases, tables
> and columns and thus the backend db can get swamped really bad when there are
> thousands of tables with hundreds of columns.
> The output of listTableAllColumnGrants() is not used completely so all the
> columns the PM has tried to retrieves anyway goes waste.
> Fix this by using direct SQL to fetch column privileges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22514) HiveProtoLoggingHook might leak memory
[
https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977932#comment-16977932
]
Hive QA commented on HIVE-22514:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986253/HIVE-22514.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17705 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
org.apache.hive.service.server.TestHS2HttpServerPam.org.apache.hive.service.server.TestHS2HttpServerPam
(batchId=240)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19500/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19500/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19500/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986253 - PreCommit-HIVE-Build
> HiveProtoLoggingHook might leak memory
> --
>
> Key: HIVE-22514
> URL: https://issues.apache.org/jira/browse/HIVE-22514
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24
> PM.png
>
>
> HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer
> tasks and to periodically handle rollover. The builtin
> ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced
> from the outside. If log events are generated at a very fast rate this queue
> can grow large.
> !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22514) HiveProtoLoggingHook might leak memory
[ https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977921#comment-16977921 ] Slim Bouguerra commented on HIVE-22514: --- +1 pending on tests thanks [~amagyar] > HiveProtoLoggingHook might leak memory > -- > > Key: HIVE-22514 > URL: https://issues.apache.org/jira/browse/HIVE-22514 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24 > PM.png > > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22514) HiveProtoLoggingHook might leak memory
[
https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977900#comment-16977900
]
Hive QA commented on HIVE-22514:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 9 unchanged - 0 fixed
= 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19500/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19500/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19500/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveProtoLoggingHook might leak memory
> --
>
> Key: HIVE-22514
> URL: https://issues.apache.org/jira/browse/HIVE-22514
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24
> PM.png
>
>
> HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer
> tasks and to periodically handle rollover. The builtin
> ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced
> from the outside. If log events are generated at a very fast rate this queue
> can grow large.
> !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Status: Open (was: Patch Available) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.2.patch, HIVE-22510.3.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: HIVE-22510.3.patch Status: Patch Available (was: Open) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.2.patch, HIVE-22510.3.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.9.1
[
https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977868#comment-16977868
]
Hive QA commented on HIVE-21737:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 41 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19499/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19499/yetus/whitespace-tabs.txt
|
| modules | C: serde ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19499/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade Avro to version 1.9.1
> -
>
> Key: HIVE-21737
> URL: https://issues.apache.org/jira/browse/HIVE-21737
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ismaël Mejía
>Assignee: Fokko Driesprong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch,
> 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner
> version of Avro without Jackson in the public API. Worth the update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.9.1
[
https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977875#comment-16977875
]
Hive QA commented on HIVE-21737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986256/0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 108 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column2]
(batchId=100)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column3]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column_extschema]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_alter_table_update_columns]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_charvarchar]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_comments]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_compression_enabled_native]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_date] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_native]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_decimal_old]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_deserialize_map_null]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_historical_timestamp]
(batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_joins_native]
(batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_native] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_partitioned_native]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_schema_evolution_native]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_timestamp]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_ppd_non_deterministic]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_avro_partition_uniontype]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_null]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_deterministic_expr]
(batchId=20)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update_llap_io]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=112)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_decimal]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_compression_enabled_native]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_decimal_native]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_joins_native]
(batchId=155)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=275)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroBinarySchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroBooleanSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroBytesSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroCharSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroDateSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroDecimalSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroDoubleSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroFloatSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroIntSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroListSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroLongSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroMapSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroNestedStructSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroShortSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroStringSchema
(batchId=361)
org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.createAvroStructSchema
(batchId=361)
[jira] [Commented] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977824#comment-16977824
]
Hive QA commented on HIVE-22513:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986234/HIVE-22513.0.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17703 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml
file (likely timed out) (batchId=256)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19498/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19498/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19498/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986234 - PreCommit-HIVE-Build
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22513.0.patch
>
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate a constant.
> Rather than adding a big and potentially convoluted and fragile check for
> this, I propose to prevent constant mappings to be spawned out of CASTed
> columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977791#comment-16977791
]
Hive QA commented on HIVE-22513:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19498/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19498/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22513.0.patch
>
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate
[jira] [Commented] (HIVE-22507) KeyWrapper comparator create field comparator instances at every comparison
[
https://issues.apache.org/jira/browse/HIVE-22507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977751#comment-16977751
]
Hive QA commented on HIVE-22507:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986213/HIVE-22507.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=177)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19497/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19497/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19497/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986213 - PreCommit-HIVE-Build
> KeyWrapper comparator create field comparator instances at every comparison
>
>
> Key: HIVE-22507
> URL: https://issues.apache.org/jira/browse/HIVE-22507
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22507.1.patch
>
>
> TopNKeyOperator.KeyWrapperComparator uses separate comparators for each field
> of the keys. Every time when TopNKeyOperator.KeyWrapperComparator.compare is
> called a new instance of the field operators are created.
> The field comparators should be created prior any comparison. Probably in the
> constructor of KeyWrapperComparator or during initializeOp.
> https://github.com/apache/hive/blob/fc81b8909b1a0e6aa15900387a98bccf38ae2247/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L964
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Patch Available (was: Open) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: HIVE-22505.3.patch > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Status: Open (was: Patch Available) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: HIVE-22505.2.patch, HIVE-22505.3.patch, > HIVE-22505.patch, query_error.out, query_vector_explain.out, vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977717#comment-16977717
]
Jesus Camacho Rodriguez commented on HIVE-22476:
+1
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Status: Patch Available (was: Open)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Attachment: HIVE-22476.7.patch
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch, HIVE-22476.7.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Attachment: (was: HIVE-22476.7.patch)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Status: Open (was: Patch Available)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22507) KeyWrapper comparator create field comparator instances at every comparison
[
https://issues.apache.org/jira/browse/HIVE-22507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977685#comment-16977685
]
Hive QA commented on HIVE-22507:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
39s{color} | {color:red} ql generated 1 new + 1539 unchanged - 0 fixed = 1540
total (was 1539) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.exec.TopNKeyOperator$KeyWrapperComparator
implements Comparator but not Serializable At
TopNKeyOperator.java:Serializable At TopNKeyOperator.java:[lines 63-80] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19497/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19497/yetus/new-findbugs-ql.html
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19497/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> KeyWrapper comparator create field comparator instances at every comparison
>
>
> Key: HIVE-22507
> URL: https://issues.apache.org/jira/browse/HIVE-22507
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22507.1.patch
>
>
> TopNKeyOperator.KeyWrapperComparator uses separate comparators for each field
> of the keys. Every time when TopNKeyOperator.KeyWrapperComparator.compare is
> called a new instance of the field operators are created.
> The field comparators should be created prior any comparison. Probably in the
> constructor of
[jira] [Commented] (HIVE-22514) HiveProtoLoggingHook might leak memory
[ https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977679#comment-16977679 ] Attila Magyar commented on HIVE-22514: -- cc: [~rzhappy] > HiveProtoLoggingHook might leak memory > -- > > Key: HIVE-22514 > URL: https://issues.apache.org/jira/browse/HIVE-22514 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24 > PM.png > > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-5208) Provide an easier way to capture DEBUG logging
[ https://issues.apache.org/jira/browse/HIVE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977656#comment-16977656 ] Piyush Patel commented on HIVE-5208: [~leftyl] Although this seems possible solution, having straight forward solution for developers could be appreciated. In second option, it is upto Sysadmins to configure logging on Hive server2. In the first case, I get HiveServer2 Kerberos principal or keytab is not correctly configured. I am actually not sure if we can pass principal and keytab as parameters to Hiveserver2 cli. > Provide an easier way to capture DEBUG logging > -- > > Key: HIVE-5208 > URL: https://issues.apache.org/jira/browse/HIVE-5208 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 0.11.0 >Reporter: Harsh J >Priority: Minor > > Capturing debug logging for troubleshooting is painful in Hive today: > 1. It doesn't log anywhere by default. > 2. We need to add a long "-hiveconf hive.root.logger=DEBUG,console" to the > Hive CLI just to enable the debug flag, or set an equivalent env-var > appropriately. > I suggest we make this simpler via either one of the below: > 1. Provide a wrapped binary, "hive-debug", so folks can simply re-run > the "hive-debug" command and re-run their query and capture an output. This > could also write to a pre-designated $PWD file. > 2. Provide a simpler switch, such as "-verbose" that automatically > toggles the flag instead, much like what Beeline does today already. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22369) Handle HiveTableFunctionScan at return path
[ https://issues.apache.org/jira/browse/HIVE-22369?focusedWorklogId=346094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346094 ] ASF GitHub Bot logged work on HIVE-22369: - Author: ASF GitHub Bot Created on: 19/Nov/19 17:03 Start Date: 19/Nov/19 17:03 Worklog Time Spent: 10m Work Description: miklosgergely commented on issue #845: HIVE-22369 Handle HiveTableFunctionScan at return path URL: https://github.com/apache/hive/pull/845#issuecomment-555606077 Added unit tests for the modifications. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 346094) Time Spent: 50m (was: 40m) > Handle HiveTableFunctionScan at return path > --- > > Key: HIVE-22369 > URL: https://issues.apache.org/jira/browse/HIVE-22369 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22369.01.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The > [optimizedOptiqPlan|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1573] > at CalcitePlanner.getOptimizedHiveOPDag is ultimately generated by > CalcitePlanner.internalGenSelectLogicalPlan, which may either provide a > [HiveProject|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L4831] > or a > [HiveTableFunctionScan|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L4776]. > When HiveCalciteUtil.getTopLevelSelect is invoked on this it is looking for > a > [HiveProject|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java#L633] > node in the tree, which if won't find in case of a HiveTableFunctionScan was > returned. This is why TestNewGetSplitsFormat is failing with return path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22369) Handle HiveTableFunctionScan at return path
[
https://issues.apache.org/jira/browse/HIVE-22369?focusedWorklogId=346093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346093
]
ASF GitHub Bot logged work on HIVE-22369:
-
Author: ASF GitHub Bot
Created on: 19/Nov/19 17:02
Start Date: 19/Nov/19 17:02
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #845:
HIVE-22369 Handle HiveTableFunctionScan at return path
URL: https://github.com/apache/hive/pull/845#discussion_r348050730
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
##
@@ -186,12 +193,67 @@ OpAttr dispatch(RelNode rn) throws SemanticException {
return visit((HiveSortExchange) rn);
} else if (rn instanceof HiveAggregate) {
return visit((HiveAggregate) rn);
+} else if (rn instanceof HiveTableFunctionScan) {
+ return visit((HiveTableFunctionScan) rn);
}
LOG.error(rn.getClass().getCanonicalName() + "operator translation not
supported"
+ " yet in return path.");
return null;
}
+ private OpAttr visit(HiveTableFunctionScan scanRel) throws SemanticException
{
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Translating operator rel#" + scanRel.getId() + ":"
+ + scanRel.getRelTypeName() + " with row type: [" +
scanRel.getRowType() + "]");
+}
+
+RexCall call = (RexCall)scanRel.getCall();
+
+String functionName = call.getOperator().getName();
+FunctionInfo fi = FunctionRegistry.getFunctionInfo(functionName);
+GenericUDTF genericUDTF = fi.getGenericUDTF();
+
+RowResolver rowResolver = new RowResolver();
+List fieldNames = new
ArrayList<>(scanRel.getRowType().getFieldNames());
+List exprNames = new ArrayList<>(fieldNames);
+List exprCols = new ArrayList<>();
+Map colExprMap = new HashMap<>();
+for (int pos = 0; pos < call.getOperands().size(); pos++) {
+ ExprNodeConverter converter = new
ExprNodeConverter(SemanticAnalyzer.DUMMY_TABLE, fieldNames.get(pos),
+ scanRel.getRowType(), scanRel.getRowType(),
((HiveTableScan)scanRel.getInput(0)).getPartOrVirtualCols(),
+ scanRel.getCluster().getTypeFactory(), true);
+ ExprNodeDesc exprCol = call.getOperands().get(pos).accept(converter);
+ colExprMap.put(exprNames.get(pos), exprCol);
+ exprCols.add(exprCol);
+
+ ColumnInfo columnInfo = new ColumnInfo(fieldNames.get(pos),
exprCol.getWritableObjectInspector(), null, false);
+ rowResolver.put(columnInfo.getTabAlias(), columnInfo.getAlias(),
columnInfo);
+}
+
+QB qb = new QB(semanticAnalyzer.getQB().getId(), nextAlias(), true);
+qb.getMetaData().setSrcForAlias(SemanticAnalyzer.DUMMY_TABLE,
semanticAnalyzer.getDummyTable());
+TableScanOperator op = (TableScanOperator)
semanticAnalyzer.genTablePlan(SemanticAnalyzer.DUMMY_TABLE, qb);
Review comment:
QB will be removed completely? I'll check out how we could integrate this to
`visit(HiveTableScan scanRel)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 346093)
Time Spent: 40m (was: 0.5h)
> Handle HiveTableFunctionScan at return path
> ---
>
> Key: HIVE-22369
> URL: https://issues.apache.org/jira/browse/HIVE-22369
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22369.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The
> [optimizedOptiqPlan|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1573]
> at CalcitePlanner.getOptimizedHiveOPDag is ultimately generated by
> CalcitePlanner.internalGenSelectLogicalPlan, which may either provide a
> [HiveProject|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L4831]
> or a
> [HiveTableFunctionScan|https://github.com/apache/hive/blob/5c91d324f22c2ae47e234e76a9bc5ee1a71e6a70/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L4776].
> When HiveCalciteUtil.getTopLevelSelect is invoked on this it is looking for
> a
>
[jira] [Work logged] (HIVE-22369) Handle HiveTableFunctionScan at return path
[
https://issues.apache.org/jira/browse/HIVE-22369?focusedWorklogId=346092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346092
]
ASF GitHub Bot logged work on HIVE-22369:
-
Author: ASF GitHub Bot
Created on: 19/Nov/19 17:02
Start Date: 19/Nov/19 17:02
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #845:
HIVE-22369 Handle HiveTableFunctionScan at return path
URL: https://github.com/apache/hive/pull/845#discussion_r348049562
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
##
@@ -186,12 +193,67 @@ OpAttr dispatch(RelNode rn) throws SemanticException {
return visit((HiveSortExchange) rn);
} else if (rn instanceof HiveAggregate) {
return visit((HiveAggregate) rn);
+} else if (rn instanceof HiveTableFunctionScan) {
+ return visit((HiveTableFunctionScan) rn);
}
LOG.error(rn.getClass().getCanonicalName() + "operator translation not
supported"
+ " yet in return path.");
return null;
}
+ private OpAttr visit(HiveTableFunctionScan scanRel) throws SemanticException
{
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Translating operator rel#" + scanRel.getId() + ":"
+ + scanRel.getRelTypeName() + " with row type: [" +
scanRel.getRowType() + "]");
+}
+
+RexCall call = (RexCall)scanRel.getCall();
+
+String functionName = call.getOperator().getName();
+FunctionInfo fi = FunctionRegistry.getFunctionInfo(functionName);
+GenericUDTF genericUDTF = fi.getGenericUDTF();
+
+RowResolver rowResolver = new RowResolver();
+List fieldNames = new
ArrayList<>(scanRel.getRowType().getFieldNames());
+List exprNames = new ArrayList<>(fieldNames);
+List exprCols = new ArrayList<>();
+Map colExprMap = new HashMap<>();
+for (int pos = 0; pos < call.getOperands().size(); pos++) {
+ ExprNodeConverter converter = new
ExprNodeConverter(SemanticAnalyzer.DUMMY_TABLE, fieldNames.get(pos),
+ scanRel.getRowType(), scanRel.getRowType(),
((HiveTableScan)scanRel.getInput(0)).getPartOrVirtualCols(),
+ scanRel.getCluster().getTypeFactory(), true);
+ ExprNodeDesc exprCol = call.getOperands().get(pos).accept(converter);
+ colExprMap.put(exprNames.get(pos), exprCol);
+ exprCols.add(exprCol);
+
+ ColumnInfo columnInfo = new ColumnInfo(fieldNames.get(pos),
exprCol.getWritableObjectInspector(), null, false);
+ rowResolver.put(columnInfo.getTabAlias(), columnInfo.getAlias(),
columnInfo);
+}
+
+QB qb = new QB(semanticAnalyzer.getQB().getId(), nextAlias(), true);
+qb.getMetaData().setSrcForAlias(SemanticAnalyzer.DUMMY_TABLE,
semanticAnalyzer.getDummyTable());
+TableScanOperator op = (TableScanOperator)
semanticAnalyzer.genTablePlan(SemanticAnalyzer.DUMMY_TABLE, qb);
+op.getConf().setRowLimit(1);
+qb.addAlias(SemanticAnalyzer.DUMMY_TABLE);
+qb.setTabAlias(SemanticAnalyzer.DUMMY_TABLE, SemanticAnalyzer.DUMMY_TABLE);
+
+Operator output = OperatorFactory.getAndMakeChild(new
SelectDesc(exprCols, fieldNames, false),
+new RowSchema(rowResolver.getRowSchema()), op);
+output.setColumnExprMap(colExprMap);
+semanticAnalyzer.putOpInsertMap(output, rowResolver);
+
+Operator funcOp = semanticAnalyzer.genUDTFPlan(genericUDTF, null,
fieldNames, qb, output, false);
Review comment:
Moving `genUDTFPlan` to `HiveOpConverter` would mean that it should be
copied from the `SemanticAnalyzer`, and having a duplication before the old one
get's removed? It's ok for me, but we should also consider that
`HiveOpConverter` is already 1300 lines long, and it wouldn't be wise to create
another monster class. So I'm ok with moving it here, but then soon we should
have a jira for creating a better design here too, and cutting
`HiveOpConverter` to pieces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 346092)
Time Spent: 40m (was: 0.5h)
> Handle HiveTableFunctionScan at return path
> ---
>
> Key: HIVE-22369
> URL: https://issues.apache.org/jira/browse/HIVE-22369
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22369.01.patch
>
> Time Spent: 40m
> Remaining
[jira] [Commented] (HIVE-22481) Expose null sort order at default level
[
https://issues.apache.org/jira/browse/HIVE-22481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977616#comment-16977616
]
Hive QA commented on HIVE-22481:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
33s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19496/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19496/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose null sort order at default level
> ---
>
> Key: HIVE-22481
> URL: https://issues.apache.org/jira/browse/HIVE-22481
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-22481.1.patch, HIVE-22481.1.patch
>
>
> ReduceSinkDesc
> https://github.com/apache/hive/blob/91f9ff5648854c487860a79b82450538fbcf6c48/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java#L410
> and TopNKeyDesc
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22435:
--
Status: Open (was: Patch Available)
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch, HIVE-22435.7.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
> [0:key._col0, 1:key._col1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:538)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.TopNKeyOperator.initializeOp(TopNKeyOperator.java:106)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.initializeOp(VectorTopNKeyOperator.java:71)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1572454329409_0001_9_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
[jira] [Updated] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22435:
--
Attachment: HIVE-22435.7.patch
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch, HIVE-22435.7.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
> [0:key._col0, 1:key._col1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:538)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.TopNKeyOperator.initializeOp(TopNKeyOperator.java:106)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.initializeOp(VectorTopNKeyOperator.java:71)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1572454329409_0001_9_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
>
[jira] [Updated] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22435:
--
Status: Patch Available (was: Open)
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch, HIVE-22435.7.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
> [0:key._col0, 1:key._col1]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:538)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.TopNKeyOperator.initializeOp(TopNKeyOperator.java:106)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.initializeOp(VectorTopNKeyOperator.java:71)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1572454329409_0001_9_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: (was: 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: (was: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch Status: Patch Available (was: Open) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HIVE-21737: Status: Open (was: Patch Available) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977589#comment-16977589
]
Hive QA commented on HIVE-22435:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986206/HIVE-22435.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17699 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniLlapArrow - did not produce a TEST-*.xml file (likely timed
out) (batchId=285)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testActiveSessionTimeMetrics
(batchId=239)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19495/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19495/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19495/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986206 - PreCommit-HIVE-Build
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
>
[jira] [Updated] (HIVE-22512) Use direct SQL to fetch column privileges in refreshPrivileges
[ https://issues.apache.org/jira/browse/HIVE-22512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22512: -- Attachment: HIVE-22512.01.patch Status: Patch Available (was: Open) > Use direct SQL to fetch column privileges in refreshPrivileges > -- > > Key: HIVE-22512 > URL: https://issues.apache.org/jira/browse/HIVE-22512 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Attachments: HIVE-22512.01.patch > > > refreshPrivileges() calls listTableAllColumnGrants() to fetch the column > level privileges. The later function retrieves the individual column objects > by firing one query per column privilege object, thus causing the backend db > to be swamped by these queries when PrivilegeSynchronizer is run. > PrivilegeSynchronizer synchronizes privileges of all the databases, tables > and columns and thus the backend db can get swamped really bad when there are > thousands of tables with hundreds of columns. > The output of listTableAllColumnGrants() is not used completely so all the > columns the PM has tried to retrieves anyway goes waste. > Fix this by using direct SQL to fetch column privileges. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22514) HiveProtoLoggingHook might leak memory
[ https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22514: - Status: Patch Available (was: Open) > HiveProtoLoggingHook might leak memory > -- > > Key: HIVE-22514 > URL: https://issues.apache.org/jira/browse/HIVE-22514 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24 > PM.png > > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22514) HiveProtoLoggingHook might leak memory
[ https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22514: - Attachment: HIVE-22514.1.patch > HiveProtoLoggingHook might leak memory > -- > > Key: HIVE-22514 > URL: https://issues.apache.org/jira/browse/HIVE-22514 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22514.1.patch, Screen Shot 2019-11-18 at 2.19.24 > PM.png > > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977548#comment-16977548 ] Ismaël Mejía commented on HIVE-21737: - Yes ! Question [~belugabehr] is there a way to trigger the patch again? I did not save the logs tests to see what went wrong. And extra question is there an exact command line invocation that I can use to reproduce the CI run locally? > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977549#comment-16977549
]
Hive QA commented on HIVE-22435:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 6 new + 437 unchanged - 2
fixed = 443 total (was 439) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19495/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19495/yetus/diff-checkstyle-ql.txt
|
| modules | C: serde ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19495/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch,
> HIVE-22435.5.patch, HIVE-22435.6.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR]
[jira] [Assigned] (HIVE-22514) HiveProtoLoggingHook might leak memory
[ https://issues.apache.org/jira/browse/HIVE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar reassigned HIVE-22514: > HiveProtoLoggingHook might leak memory > -- > > Key: HIVE-22514 > URL: https://issues.apache.org/jira/browse/HIVE-22514 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: Screen Shot 2019-11-18 at 2.19.24 PM.png > > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > !Screen Shot 2019-11-18 at 2.19.24 PM.png|width=650,height=101! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: (was: 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: (was: 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated HIVE-21737: Attachment: 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977545#comment-16977545 ] Fokko Driesprong commented on HIVE-21737: - [~belugabehr] I have very little time at the moment. I think [~iemejia] will give it another shot. > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch, > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21737) Upgrade Avro to version 1.9.1
[ https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated HIVE-21737: Attachment: (was: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.1.patch) > Upgrade Avro to version 1.9.1 > - > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: > 0002-HIVE-21737-Use-NULL_DEFAULT_VALUE-for-Avro-fields-in.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Avro 1.9.0 was released recently. It brings a lot of fixes including a leaner > version of Avro without Jackson in the public API. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22379) Reduce db lookups during dynamic partition loading
[
https://issues.apache.org/jira/browse/HIVE-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977531#comment-16977531
]
Hive QA commented on HIVE-22379:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986198/HIVE-22379.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.dynamicPartitioningInsert
(batchId=249)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.schemaEvolutionAddColDynamicPartitioningUpdate
(batchId=249)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19494/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19494/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19494/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986198 - PreCommit-HIVE-Build
> Reduce db lookups during dynamic partition loading
> --
>
> Key: HIVE-22379
> URL: https://issues.apache.org/jira/browse/HIVE-22379
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: performance
> Attachments: HIVE-22379.1.patch, HIVE-22379.2.patch,
> HIVE-22379.3.patch, HIVE-22379.4.patch
>
>
> {{HiveAlterHandler::alterPartitions}} could lookup all partition details via
> single call instead of multiple lookups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22379) Reduce db lookups during dynamic partition loading
[
https://issues.apache.org/jira/browse/HIVE-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977495#comment-16977495
]
Hive QA commented on HIVE-22379:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 378 unchanged - 3 fixed = 383 total (was 381) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 120
unchanged - 8 fixed = 130 total (was 128) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19494/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19494/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19494/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19494/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce db lookups during dynamic partition loading
> --
>
> Key: HIVE-22379
> URL: https://issues.apache.org/jira/browse/HIVE-22379
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: performance
> Attachments: HIVE-22379.1.patch, HIVE-22379.2.patch,
> HIVE-22379.3.patch, HIVE-22379.4.patch
>
>
> {{HiveAlterHandler::alterPartitions}} could lookup all partition details via
> single call
[jira] [Commented] (HIVE-22486) Send only accessed columns for masking policies request
[
https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977470#comment-16977470
]
Hive QA commented on HIVE-22486:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986185/HIVE-22486.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17707 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19493/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19493/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19493/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986185 - PreCommit-HIVE-Build
> Send only accessed columns for masking policies request
> ---
>
> Key: HIVE-22486
> URL: https://issues.apache.org/jira/browse/HIVE-22486
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch,
> HIVE-22486.patch
>
>
> Currently, we send all columns for masking request, even if they are not
> accessed by the given query. We could send only those columns for which the
> masking policy will be necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22513:
--
Status: Patch Available (was: Open)
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22513.0.patch
>
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate a constant.
> Rather than adding a big and potentially convoluted and fragile check for
> this, I propose to prevent constant mappings to be spawned out of CASTed
> columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22513:
--
Attachment: HIVE-22513.0.patch
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22513.0.patch
>
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate a constant.
> Rather than adding a big and potentially convoluted and fragile check for
> this, I propose to prevent constant mappings to be spawned out of CASTed
> columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977457#comment-16977457 ] Denys Kuzmenko commented on HIVE-22327: --- [~maheshk114], I have addressed all your comments in a latest patch. Could you please take a loot. Thanks! > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.1.patch, HIVE-22327.10.patch, > HIVE-22327.2.patch, HIVE-22327.3.patch, HIVE-22327.4.patch, > HIVE-22327.5.patch, HIVE-22327.6.patch, HIVE-22327.7.patch, > HIVE-22327.8.patch, HIVE-22327.9.patch > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22513:
--
Description:
This issue happens if CBO is disabled.
We should not be propagating constants if the corresponding ExprNodeColumnDesc
instance is wrapped inside a CAST operator as casting might truncate
information from the original column.
This can happen if we're using CAST in a WHERE clause, which will cause the
projected columns to be replaced in a SELECT operator. Their new value will be
the result of casting which could be a different value compared to that in the
original column:
{code:java}
set hive.cbo.enable=false;
set hive.fetch.task.conversion=more; --just for testing convenience
create table testtb (id string);
insert into testtb values('2019-11-05 01:01:11');
select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
'2019-11-0';
+++
| id | _c1 |
+++
| 2019-11-0 | 2019-11-0 |
+++
1 row selected (0.168 seconds)
-- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
As to what types of casting (from and where types) cause information loss it's
hard to properly keep track of, and I don't think it should be taken into
consideration when deciding whether or not to propagate a constant. Rather than
adding a big and potentially convoluted and fragile check for this, I propose
to prevent constant mappings to be spawned out of CASTed columns.
was:
This issue happens if CBO is disabled.
We should not be propagating constants if the corresponding ExprNodeColumnDesc
instance is wrapped inside a CAST operator as casting might truncate
information from the original column.
This can happen if we're using CAST in a WHERE clause, which will cause the
projected columns to be replaced in a SELECT operator. Their new value will be
the result of casting which could be a different value compared to that in the
original column:
{code:java}
set hive.cbo.enable=false;
set hive.fetch.task.conversion=more; --just for testing convenience
create table testtb (id string);
insert into testtb values('2019-11-05 01:01:11');
select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
'2019-11-0';
+++
| id | _c1 |
+++
| 2019-11-0 | 2019-11-0 |
+++
1 row selected (0.168 seconds)
-- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
As to what types of casting (from and where types) cause information loss it's
hard to properly keep track of, and I don't think it should be taken into
consideration when deciding whether or not to propagate a constant. Rather than
adding a big and potentially convoluted and fragile check for this, I propose
to prevent constant mappings to be spawned out of CASTed columns.
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate a constant.
> Rather than adding a big and potentially convoluted and fragile check for
> this, I propose to prevent constant mappings to be spawned out of CASTed
> columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22513) Constant propagation of casted column in filter ops can cause incorrect results
[
https://issues.apache.org/jira/browse/HIVE-22513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita reassigned HIVE-22513:
-
> Constant propagation of casted column in filter ops can cause incorrect
> results
> ---
>
> Key: HIVE-22513
> URL: https://issues.apache.org/jira/browse/HIVE-22513
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> This issue happens if CBO is disabled.
> We should not be propagating constants if the corresponding
> ExprNodeColumnDesc instance is wrapped inside a CAST operator as casting
> might truncate information from the original column.
> This can happen if we're using CAST in a WHERE clause, which will cause the
> projected columns to be replaced in a SELECT operator. Their new value will
> be the result of casting which could be a different value compared to that in
> the original column:
> {code:java}
> set hive.cbo.enable=false;
> set hive.fetch.task.conversion=more; --just for testing convenience
> create table testtb (id string);
> insert into testtb values('2019-11-05 01:01:11');
> select id, CAST(id AS VARCHAR(10)) from testtb where CAST(id AS VARCHAR(9)) =
> '2019-11-0';
> +++
> | id | _c1 |
> +++
> | 2019-11-0 | 2019-11-0 |
> +++
> 1 row selected (0.168 seconds)
> -- VS expected: 2019-11-05 01:01:11 | 2019-11-05 {code}
> As to what types of casting (from and where types) cause information loss
> it's hard to properly keep track of, and I don't think it should be taken
> into consideration when deciding whether or not to propagate a constant.
> Rather than adding a big and potentially convoluted and fragile check for
> this, I propose to prevent constant mappings to be spawned out of CASTed
> columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22486) Send only accessed columns for masking policies request
[
https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977424#comment-16977424
]
Hive QA commented on HIVE-22486:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 1 new + 593 unchanged - 0
fixed = 594 total (was 593) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19493/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19493/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19493/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Send only accessed columns for masking policies request
> ---
>
> Key: HIVE-22486
> URL: https://issues.apache.org/jira/browse/HIVE-22486
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch,
> HIVE-22486.patch
>
>
> Currently, we send all columns for masking request, even if they are not
> accessed by the given query. We could send only those columns for which the
> masking policy will be necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22512) Use direct SQL to fetch column privileges in refreshPrivileges
[ https://issues.apache.org/jira/browse/HIVE-22512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat reassigned HIVE-22512: - > Use direct SQL to fetch column privileges in refreshPrivileges > -- > > Key: HIVE-22512 > URL: https://issues.apache.org/jira/browse/HIVE-22512 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > > refreshPrivileges() calls listTableAllColumnGrants() to fetch the column > level privileges. The later function retrieves the individual column objects > by firing one query per column privilege object, thus causing the backend db > to be swamped by these queries when PrivilegeSynchronizer is run. > PrivilegeSynchronizer synchronizes privileges of all the databases, tables > and columns and thus the backend db can get swamped really bad when there are > thousands of tables with hundreds of columns. > The output of listTableAllColumnGrants() is not used completely so all the > columns the PM has tried to retrieves anyway goes waste. > Fix this by using direct SQL to fetch column privileges. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20587) Reduce phase could not be completed on MapReduce engine
[
https://issues.apache.org/jira/browse/HIVE-20587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977412#comment-16977412
]
Abhishek commented on HIVE-20587:
-
it can be tuned through by editing vi
/usr/local/hadoop-3.1.3/etc/hadoop/mapred-site.xml
mapreduce.map.memory.mb
4096
mapreduce.reduce.memory.mb
4096
rest could be fine
> Reduce phase could not be completed on MapReduce engine
> ---
>
> Key: HIVE-20587
> URL: https://issues.apache.org/jira/browse/HIVE-20587
> Project: Hive
> Issue Type: Bug
>Reporter: Mykhailo Kysliuk
>Priority: Minor
>
> Steps to reproduce:
> {code}
> CREATE TABLE IF NOT EXISTS `test`(
> `account_id` bigint,
> `subscription_id` bigint,
> `previous_subscription_id` bigint,
> `subscription_effective_from_dt` date,
> `subscription_effective_to_dt` date,
> `order_completed_dtm` timestamp,
> `subscription_status` string) STORED AS ORC;
> insert into test values (1, 1, 1, NULL, NULL, NULL, 'ss');
> SELECT
> account_id,
> subscription_id,
> previous_subscription_id,
> subscription_status,
> subscription_effective_from_dt,
> subscription_effective_to_dt
> FROM
> (
> SELECT
> DISTINCT account_id,
> subscription_id,
> previous_subscription_id,
> subscription_status,
> subscription_effective_from_dt,
> subscription_effective_to_dt,
> order_completed_dtm,
> ROW_NUMBER()OVER(PARTITION BY ACCOUNT_ID) as r1
> FROM test
> WHERE account_id=1
> ) AS RTE where r1=1;
> {code}
> Expected result:
> {code}
> OK
> 1 1 1 ss NULLNULL
> {code}
> Actual result:
> {code}
> Starting Job = job_1537276416358_0006, Tracking URL =
> https://node10.cluster.com:8090/proxy/application_1537276416358_0006/
> Kill Command = /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop job -kill
> job_1537276416358_0006
> Hadoop job information for Stage-1: number of mappers: 1; number of reducers:
> 1
> 2018-09-18 13:21:00,754 Stage-1 map = 0%, reduce = 0%
> 2018-09-18 13:21:04,913 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:22:05,152 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:23:05,540 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:24:06,532 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:25:07,477 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:26:07,914 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:27:08,021 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:28:08,935 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:29:08,970 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:30:09,054 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:31:09,965 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:32:10,922 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:33:10,964 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:34:11,005 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:35:11,978 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:36:13,057 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:37:13,208 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:38:13,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:39:13,719 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:40:13,729 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:41:14,103 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:42:14,160 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:43:14,241 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:44:14,345 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:45:15,353 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:46:15,546 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:47:15,608 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:48:16,481 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:49:16,522 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:50:16,774 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:51:17,784 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:52:17,838 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3
> sec
> 2018-09-18 13:53:17,868 Stage-1 map = 100%, reduce = 0%,
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[
https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977407#comment-16977407
]
Hive QA commented on HIVE-22510:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986182/HIVE-22510.2.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 55 failed/errored test(s), 17713 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_not]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal64_mul_longcolumn]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions_ramesh]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_udf_trunc]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_10]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_3]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_not]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=64)
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[llap_0]
(batchId=296)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=188)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reuse_scratchcols]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_10]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_13]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_17]
(batchId=188)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_1]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_3]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_not]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_1]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_not]
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_10]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17]
(batchId=154)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_1]
(batchId=141)
[jira] [Assigned] (HIVE-22511) Fix case of Month token in datetime to string conversion
[
https://issues.apache.org/jira/browse/HIVE-22511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage reassigned HIVE-22511:
Assignee: Karen Coppage
> Fix case of Month token in datetime to string conversion
>
>
> Key: HIVE-22511
> URL: https://issues.apache.org/jira/browse/HIVE-22511
> Project: Hive
> Issue Type: Bug
>Reporter: Gabor Kaszab
>Assignee: Karen Coppage
>Priority: Major
>
> Currently Hive doesn't allow month tokens with weird spelling like 'MONth',
> 'mONTH' etc. However, Oracle does and Hive should follow that approach.
> The rules:
> - If the first letter is lowercase then the output is lowercase: 'mONTH' ->
> 'may'
> - If the first two letters are uppercase then the output is uppercase:
> 'MOnth' -> 'MAY'
> - If the first letter is uppercase and the second is lowercase then the
> output is capitalized: 'Month' -> 'May'.
> Oracle:
> {code:java}
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'MOnth') from
> DUAL;
> MAY 2019
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'mONTH') from
> DUAL;
> may 2019
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'MoNTH') from
> DUAL;
> May 2019
> {code}
> Please check the same for 'Name of the day' tokens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22511) Fix case of Month token in datetime to string conversion
[
https://issues.apache.org/jira/browse/HIVE-22511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977366#comment-16977366
]
Gabor Kaszab commented on HIVE-22511:
-
[~klcopp]
> Fix case of Month token in datetime to string conversion
>
>
> Key: HIVE-22511
> URL: https://issues.apache.org/jira/browse/HIVE-22511
> Project: Hive
> Issue Type: Bug
>Reporter: Gabor Kaszab
>Priority: Major
>
> Currently Hive doesn't allow month tokens with weird spelling like 'MONth',
> 'mONTH' etc. However, Oracle does and Hive should follow that approach.
> The rules:
> - If the first letter is lowercase then the output is lowercase: 'mONTH' ->
> 'may'
> - If the first two letters are uppercase then the output is uppercase:
> 'MOnth' -> 'MAY'
> - If the first letter is uppercase and the second is lowercase then the
> output is capitalized: 'Month' -> 'May'.
> Oracle:
> {code:java}
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'MOnth') from
> DUAL;
> MAY 2019
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'mONTH') from
> DUAL;
> may 2019
> select to_char(to_timestamp('2019-05-10', '-MM-DD'), 'MoNTH') from
> DUAL;
> May 2019
> {code}
> Please check the same for 'Name of the day' tokens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[
https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977356#comment-16977356
]
Hive QA commented on HIVE-22510:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 1539 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 3 new + 794 unchanged - 1
fixed = 797 total (was 795) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19492/dev-support/hive-personality.sh
|
| git revision | master / 44697f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19492/yetus/diff-checkstyle-ql.txt
|
| modules | C: vector-code-gen ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19492/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support decimal64 operations for column operands with different scales
> --
>
> Key: HIVE-22510
> URL: https://issues.apache.org/jira/browse/HIVE-22510
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22510.2.patch
>
>
> Right now, if the operands on the decimal64 operations are columns with
> different scales, then we do not use the decimal64 vectorized version and
> fall back to HiveDecimal vectorized version of the operator. In this Jira, we
> will check if we can use decimal64 vectorized version, even if the scales are
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[
https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977326#comment-16977326
]
Hive QA commented on HIVE-22505:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12986181/HIVE-22505.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth
(batchId=289)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19491/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19491/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19491/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12986181 - PreCommit-HIVE-Build
> ClassCastException caused by wrong Vectorized operator selection
>
>
> Key: HIVE-22505
> URL: https://issues.apache.org/jira/browse/HIVE-22505
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22505.2.patch, HIVE-22505.patch, query_error.out,
> query_vector_explain.out, vectorized_join.q
>
>
> VectorMapJoinOuterFilteredOperator does not currently support full outer
> joins but using the current Vectorizer logic it can be selected when a there
> is a filter involved. This can make queries fail with ClassCastException when
> their data and metadata in the VectorMapJoinOuterFilteredOperator do not
> match.
> The query attached demonstrates the issue and the log attached shows the
> java.lang.ClassCastException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977325#comment-16977325
]
David Lavati edited comment on HIVE-21198 at 11/19/19 10:02 AM:
Merged by [~mgergely] as
[44697f0aa766a8329|https://github.com/apache/hive/commit/44697f0aa766a8329a599572013e9ba0a6750637].
Thank you all for the help!
was (Author: dlavati):
Merged by [~mgergely] as
[44697f0aa766a8329|[https://github.com/apache/hive/commit/44697f0aa766a8329a599572013e9ba0a6750637].]
Thank you all for the help!
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.12.patch, HIVE-21198.2.patch,
> HIVE-21198.3.patch, HIVE-21198.4.patch, HIVE-21198.5.patch,
> HIVE-21198.6.patch, HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21198:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged by [~mgergely] as
[44697f0aa766a8329|[https://github.com/apache/hive/commit/44697f0aa766a8329a599572013e9ba0a6750637].]
Thank you all for the help!
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.12.patch, HIVE-21198.2.patch,
> HIVE-21198.3.patch, HIVE-21198.4.patch, HIVE-21198.5.patch,
> HIVE-21198.6.patch, HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=345879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345879
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 19/Nov/19 09:57
Start Date: 19/Nov/19 09:57
Worklog Time Spent: 10m
Work Description: dlavati commented on issue #550: HIVE-21198 Introduce a
database object reference class
URL: https://github.com/apache/hive/pull/550#issuecomment-555426283
This got meged as 44697f0aa766a8329a599572013e9ba0a6750637. Thank you for
the review, I'll include these in the followup.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345879)
Time Spent: 4h 10m (was: 4h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.12.patch, HIVE-21198.2.patch,
> HIVE-21198.3.patch, HIVE-21198.4.patch, HIVE-21198.5.patch,
> HIVE-21198.6.patch, HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=345880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-345880
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 19/Nov/19 09:57
Start Date: 19/Nov/19 09:57
Worklog Time Spent: 10m
Work Description: dlavati commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 345880)
Time Spent: 4h 20m (was: 4h 10m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.12.patch, HIVE-21198.2.patch,
> HIVE-21198.3.patch, HIVE-21198.4.patch, HIVE-21198.5.patch,
> HIVE-21198.6.patch, HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22507) KeyWrapper comparator create field comparator instances at every comparison
[ https://issues.apache.org/jira/browse/HIVE-22507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22507: -- Attachment: HIVE-22507.1.patch > KeyWrapper comparator create field comparator instances at every comparison > > > Key: HIVE-22507 > URL: https://issues.apache.org/jira/browse/HIVE-22507 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22507.1.patch > > > TopNKeyOperator.KeyWrapperComparator uses separate comparators for each field > of the keys. Every time when TopNKeyOperator.KeyWrapperComparator.compare is > called a new instance of the field operators are created. > The field comparators should be created prior any comparison. Probably in the > constructor of KeyWrapperComparator or during initializeOp. > https://github.com/apache/hive/blob/fc81b8909b1a0e6aa15900387a98bccf38ae2247/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L964 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22507) KeyWrapper comparator create field comparator instances at every comparison
[ https://issues.apache.org/jira/browse/HIVE-22507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22507: -- Status: Patch Available (was: Open) > KeyWrapper comparator create field comparator instances at every comparison > > > Key: HIVE-22507 > URL: https://issues.apache.org/jira/browse/HIVE-22507 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22507.1.patch > > > TopNKeyOperator.KeyWrapperComparator uses separate comparators for each field > of the keys. Every time when TopNKeyOperator.KeyWrapperComparator.compare is > called a new instance of the field operators are created. > The field comparators should be created prior any comparison. Probably in the > constructor of KeyWrapperComparator or during initializeOp. > https://github.com/apache/hive/blob/fc81b8909b1a0e6aa15900387a98bccf38ae2247/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L964 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
