[jira] [Updated] (HIVE-21048) Remove needless org.mortbay.jetty from hadoop exclusions
[ https://issues.apache.org/jira/browse/HIVE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21048: Attachment: (was: HIVE-21048.09.patch) > Remove needless org.mortbay.jetty from hadoop exclusions > > > Key: HIVE-21048 > URL: https://issues.apache.org/jira/browse/HIVE-21048 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21048.01.patch, HIVE-21048.02.patch, > HIVE-21048.03.patch, HIVE-21048.04.patch, HIVE-21048.05.patch, > HIVE-21048.06.patch, HIVE-21048.07.patch, HIVE-21048.08.patch, > HIVE-21048.08.patch, dep.out > > > During HIVE-20638 i found that org.mortbay.jetty exclusions from e.g. hadoop > don't take effect, as the actual groupId of jetty is org.eclipse.jetty for > most of the current projects, please find attachment (example for hive > commons project). > https://en.wikipedia.org/wiki/Jetty_(web_server)#History -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21048) Remove needless org.mortbay.jetty from hadoop exclusions
[ https://issues.apache.org/jira/browse/HIVE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787635#comment-16787635 ] Zoltan Haindrich commented on HIVE-21048: - oh my.,...9 QA runs with so much different failures... > Remove needless org.mortbay.jetty from hadoop exclusions > > > Key: HIVE-21048 > URL: https://issues.apache.org/jira/browse/HIVE-21048 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21048.01.patch, HIVE-21048.02.patch, > HIVE-21048.03.patch, HIVE-21048.04.patch, HIVE-21048.05.patch, > HIVE-21048.06.patch, HIVE-21048.07.patch, HIVE-21048.08.patch, > HIVE-21048.08.patch, HIVE-21048.09.patch, dep.out > > > During HIVE-20638 i found that org.mortbay.jetty exclusions from e.g. hadoop > don't take effect, as the actual groupId of jetty is org.eclipse.jetty for > most of the current projects, please find attachment (example for hive > commons project). > https://en.wikipedia.org/wiki/Jetty_(web_server)#History -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787625#comment-16787625
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16402/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16402/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16402/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: HIVE-21401.03.patch > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787608#comment-16787608
]
Hive QA commented on HIVE-16924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 8 new + 639 unchanged - 13
fixed = 647 total (was 652) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
2s{color} | {color:red} root: The patch generated 8 new + 647 unchanged - 13
fixed = 655 total (was 660) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} ql generated 0 new + 2256 unchanged - 2 fixed = 2256
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16400/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16400/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16400/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16400/yetus/whitespace-eol.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16400/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch,
[jira] [Commented] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[
https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787607#comment-16787607
]
Hive QA commented on HIVE-21392:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961138/HIVE-21392.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16401/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16401/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16401/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-08 06:58:24.996
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16401/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-08 06:58:24.999
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-08 06:58:26.154
+ rm -rf ../yetus_PreCommit-HIVE-Build-16401
+ mkdir ../yetus_PreCommit-HIVE-Build-16401
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16401
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16401/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: corrupt patch at line 7
fatal: corrupt patch at line 7
fatal: corrupt patch at line 7
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16401
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961138 - PreCommit-HIVE-Build
> Misconfigurations of DataNucleus log in log4j.properties
>
>
> Key: HIVE-21392
> URL: https://issues.apache.org/jira/browse/HIVE-21392
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Chen Zhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21392.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the patch of
> [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the
> DataNucleus related logging configuration from nine fine-grained loggers with
> three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth
> Jayachandran
> [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612],
> these three loggers are the top-level logger in DataNucleus, so that we
> don't need to specify other loggers for DataNucleus. However, according to
> the
> [documents|http://www.datanucleus.org/products/accessplatform/logging.html]
> and [source
> codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108]
> of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`.
> Therefore, we just need to keep the right one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787606#comment-16787606
]
Hive QA commented on HIVE-16924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961624/HIVE-16924.21.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=177)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16400/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16400/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16400/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961624 - PreCommit-HIVE-Build
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch,
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch,
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch,
> HIVE-16924.21.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: HIVE-21401.02.patch > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[
https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787567#comment-16787567
]
Bo commented on HIVE-21377:
[~pvary] Even we pass the logic for this issue, we also see the other
issue(blob-cast-error.jpg).
So, could we open the jira for the implementing the DirectSQL feature with
Oracle?
The blog-case.error-jpg, this issues is related to MetastoreDirectSqlUtils.java
{code:java}
static void setPartitionParameters(String PARTITION_PARAMS, boolean
convertMapNullsToEmptyStrings,
PersistenceManager pm, String partIds, TreeMap
partitions)
throws MetaException {
String queryText;
queryText = "select \"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\" from " +
PARTITION_PARAMS + ""
+ " where \"PART_ID\" in (" + partIds + ") and \"PARAM_KEY\" is not
null"
+ " order by \"PART_ID\" asc";
loopJoinOrderedResult(pm, partitions, queryText, 0, new
ApplyFunc() {
@Override
public void apply(Partition t, Object[] fields) {
t.putToParameters((String)fields[1], fields[2])); <<< fields[2] is
related to the WARN log
}});
// Perform conversion of null map values
for (Partition t : partitions.values())
{code}
By the way, this code is already changed in last version of Hive accordingly:
{code:java}
public void apply(Partition t, Object[] fields) {
t.putToParameters((String)fields[1], extractSqlClob(fields[2]));
}});
// Perform conversion of null map values
for (Partition t : partitions.values()) {
t.setParameters(MetaStoreServerUtils.trimMapNulls(t.getParameters(),
convertMapNullsToEmptyStrings));
{code}
By the way, could we consider to open the JIRA about the implementing DirectSQL
features for Oracle DB?
> Using Oracle as HMS DB with DirectSQL
> -
>
> Key: HIVE-21377
> URL: https://issues.apache.org/jira/browse/HIVE-21377
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Bo
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21377.01.patch, HIVE-21377.patch
>
>
> When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log
> accordingly:
> {code:java}
> 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due
> to direct SQL failure (this is not an error): Cannot extract boolean from
> column value 0 at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471)
> at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392)
> {code}
> In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean.
> But Oracle return the 0 or 1 for Boolean. So we need to modify the
> MetastoreDirectSqlUtils.java - [1]
> So, could add this snip in this code?
> {code:java}
> static Boolean extractSqlBoolean(Object value) throws MetaException {
> if (value == null) {
> return null;
> }
> if (value instanceof Boolean) {
> return (Boolean)value;
> }
> if (value instanceof Number) { // add
> try {
> return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null);
> } catch(IllegalArugmentExeception iae){
> // NOOP
> }
> if (value instanceof String) {
> try {
> return BooleanUtils.toBooleanObject((String) value, "Y", "N", null);
> } catch (IllegalArgumentException iae) {
> // NOOP
> }
> }
> throw new MetaException("Cannot extract boolean from column value " +
> value);
> }
> {code}
> [1] -
> https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21388) Constant UDF is not pushed to JDBCStorage Handler
[
https://issues.apache.org/jira/browse/HIVE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787549#comment-16787549
]
Hive QA commented on HIVE-21388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961619/HIVE-21388.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15820 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16399/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16399/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16399/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961619 - PreCommit-HIVE-Build
> Constant UDF is not pushed to JDBCStorage Handler
> -
>
> Key: HIVE-21388
> URL: https://issues.apache.org/jira/browse/HIVE-21388
> Project: Hive
> Issue Type: Improvement
> Components: CBO, StorageHandler
>Affects Versions: 4.0.0
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21388.01.patch, HIVE-21388.02.patch,
> HIVE-21388.patch
>
>
> A query involve a Hive UDF which produce a constant value does not push to
> JDBC table. Replacing the UDF with a constant push down works. Ideally, Hive
> shall first do constant folding and then push the computation.
> Here is the example:
> {code}
> explain select PRINCIPAL_NAME from sys.TBL_PRIVS where
> PRINCIPAL_NAME=current_user();
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_3]|
> | Output:["_col0"] |
> | Filter Operator [FIL_2] |
> | predicate:(_col5 = 'hrt_qa') |
> | Select Operator [SEL_1]|
> | Output:["_col5"] |
> | TableScan [TS_0] |
> | Output:["principal_name"],properties:{"hive.sql.query":"SELECT
> `tbl_grant_id`, `create_time`, `grant_option`, `grantor`, `grantor_type`,
> `principal_name`, `principal_type`, `tbl_priv`, `tbl_id`, `authorizer`\nFROM
> `TBL_PRIVS`","hive.sql.query.fieldNames":"tbl_grant_id,create_time,grant_option,grantor,grantor_type,principal_name,principal_type,tbl_priv,tbl_id,authorizer","hive.sql.query.fieldTypes":"bigint,int,int,string,string,string,string,string,bigint,string","hive.sql.query.split":"true"}
> |
> ||
> ++
> {code}
> If I replace current_user() with a constant, the predicate is pushed to table
> scan.
> Also, setting annotation deterministic=true and make initialize() return a
> ConstantObjectInspector of GenericUDFCurrentUser does not make a difference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Attachment: HIVE-21325.03.patch > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Status: Patch Available (was: Open) > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Status: Open (was: Patch Available) > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21388) Constant UDF is not pushed to JDBCStorage Handler
[
https://issues.apache.org/jira/browse/HIVE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787533#comment-16787533
]
Hive QA commented on HIVE-21388:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 58 unchanged - 2 fixed
= 60 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16399/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16399/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16399/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constant UDF is not pushed to JDBCStorage Handler
> -
>
> Key: HIVE-21388
> URL: https://issues.apache.org/jira/browse/HIVE-21388
> Project: Hive
> Issue Type: Improvement
> Components: CBO, StorageHandler
>Affects Versions: 4.0.0
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21388.01.patch, HIVE-21388.02.patch,
> HIVE-21388.patch
>
>
> A query involve a Hive UDF which produce a constant value does not push to
> JDBC table. Replacing the UDF with a constant push down works. Ideally, Hive
> shall first do constant folding and then push the computation.
> Here is the example:
> {code}
> explain select PRINCIPAL_NAME from sys.TBL_PRIVS where
> PRINCIPAL_NAME=current_user();
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_3]|
> | Output:["_col0"] |
> | Filter Operator [FIL_2] |
> | predicate:(_col5 = 'hrt_qa')
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Status: Patch Available (was: Open)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch, HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: HIVE-21264.3.patch
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch, HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Status: Open (was: Patch Available)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?focusedWorklogId=209958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209958
]
ASF GitHub Bot logged work on HIVE-21403:
-
Author: ASF GitHub Bot
Created on: 08/Mar/19 04:28
Start Date: 08/Mar/19 04:28
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #559: HIVE-21403:
Incorrect error code returned when retry bootstrap with different dump.
URL: https://github.com/apache/hive/pull/559#discussion_r263662149
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -548,6 +549,37 @@ public void
bootstrapExternalTablesDuringIncrementalPhase() throws Throwable {
);
}
+ @Test
+ public void
retryIncBootstrapExternalTablesFromDifferentDumpWithoutCleanTablesConfig()
throws Throwable {
+List dumpWithClause = Collections.singletonList(
+"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='false'"
+);
+List loadWithClause = externalTableBasePathWithClause();
+
+WarehouseInstance.Tuple tupleBootstrapWithoutExternal = primary
+.run("use " + primaryDbName)
+.run("create external table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("create table t2 as select * from t1")
+.dump(primaryDbName, null, dumpWithClause);
+
+replica.load(replicatedDbName, tupleBootstrapWithoutExternal.dumpLocation,
loadWithClause);
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209958)
Time Spent: 0.5h (was: 20m)
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch, HIVE-21403.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as target database is bootstrapped from some other path :
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/c3e5ec9e-d951-48aa-b3f4-9aeaf5e010ea.)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21403:
Status: Patch Available (was: Open)
02.patch fixed comment from Mahesh.
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch, HIVE-21403.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as target database is bootstrapped from some other path :
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/c3e5ec9e-d951-48aa-b3f4-9aeaf5e010ea.)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21403:
Attachment: HIVE-21403.02.patch
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch, HIVE-21403.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as target database is bootstrapped from some other path :
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/c3e5ec9e-d951-48aa-b3f4-9aeaf5e010ea.)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21403:
Status: Open (was: Patch Available)
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as target database is bootstrapped from some other path :
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/c3e5ec9e-d951-48aa-b3f4-9aeaf5e010ea.)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[ https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787516#comment-16787516 ] Deepak Jaiswal commented on HIVE-21408: --- [~jcamachorodriguez] [~jdere] [~vgarg] can you please review it? > Disable synthetic join predicates for non-equi joins for unintended cases > - > > Key: HIVE-21408 > URL: https://issues.apache.org/jira/browse/HIVE-21408 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-21408.1.patch > > > With support for synthetic join predicates on non-equi joins, it is important > to make sure those predicates are used only for intended purpose. Currently, > DPP and semi join reduction are not supposed to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Status: Patch Available (was: Open) Flaky test failures. Re-submitted same 04.patch. > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Attachment: HIVE-21286.04.patch > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Attachment: (was: HIVE-21286.04.patch) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Status: Open (was: Patch Available) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787503#comment-16787503
]
Teddy Choi edited comment on HIVE-21368 at 3/8/19 3:57 AM:
---
I found a commit that reverts HIVE-20315. [According to
Matt|https://issues.apache.org/jira/browse/HIVE-20315?focusedCommentId=16592355=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16592355],
"Removed DECIMAL_64 conversion avoidance changes for GROUP BY / JOIN since
they caused external test failures". It may take more than few simple changes.
was (Author: teddy.choi):
I found a commit that reverts HIVE-20315. [According to
Matt|https://issues.apache.org/jira/browse/HIVE-20315?focusedCommentId=16592355=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16592355],
the DECIMAL_64 to DECIMAL conversion was on purpose since they caused external
test failures. It may be related with more tests and take more than few simple
changes.
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787503#comment-16787503
]
Teddy Choi commented on HIVE-21368:
---
I found a commit that reverts HIVE-20315. [According to
Matt|https://issues.apache.org/jira/browse/HIVE-20315?focusedCommentId=16592355=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16592355],
the DECIMAL_64 to DECIMAL conversion was on purpose since they caused external
test failures. It may be related with more tests and take more than few simple
changes.
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-21368:
--
Comment: was deleted
(was: I found following code in Vectorizer.java. It was reverted in commit
470ba3e2835ef769f940d013acbe6c05d9208903 by McMcline in 2018-08-16 to revert
HIVE-20315. I don't know why it was reverted.
{code:java}
// For now, we don't support joins on or using DECIMAL_64.
VectorExpression[] allBigTableValueExpressions =
vContext.getVectorExpressionsUpConvertDecimal64(bigTableExprs);
{code})
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787498#comment-16787498
]
Teddy Choi commented on HIVE-21368:
---
I found following code in Vectorizer.java. It was reverted in commit
470ba3e2835ef769f940d013acbe6c05d9208903 by McMcline in 2018-08-16 to revert
HIVE-20315. I don't know why it was reverted.
{code:java}
// For now, we don't support joins on or using DECIMAL_64.
VectorExpression[] allBigTableValueExpressions =
vContext.getVectorExpressionsUpConvertDecimal64(bigTableExprs);
{code}
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[
https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787486#comment-16787486
]
Hive QA commented on HIVE-21408:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961609/HIVE-21408.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15820 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16398/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16398/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16398/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961609 - PreCommit-HIVE-Build
> Disable synthetic join predicates for non-equi joins for unintended cases
> -
>
> Key: HIVE-21408
> URL: https://issues.apache.org/jira/browse/HIVE-21408
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21408.1.patch
>
>
> With support for synthetic join predicates on non-equi joins, it is important
> to make sure those predicates are used only for intended purpose. Currently,
> DPP and semi join reduction are not supposed to use it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[
https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787469#comment-16787469
]
Hive QA commented on HIVE-21408:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16398/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16398/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable synthetic join predicates for non-equi joins for unintended cases
> -
>
> Key: HIVE-21408
> URL: https://issues.apache.org/jira/browse/HIVE-21408
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21408.1.patch
>
>
> With support for synthetic join predicates on non-equi joins, it is important
> to make sure those predicates are used only for intended purpose. Currently,
> DPP and semi join reduction are not supposed to use it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21385) Allow disabling pushdown of non-splittable computation to JDBC sources
[
https://issues.apache.org/jira/browse/HIVE-21385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787444#comment-16787444
]
Hive QA commented on HIVE-21385:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961608/HIVE-21385.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16397/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16397/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-08 02:00:47.239
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16397/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-08 02:00:47.242
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-08 02:00:49.384
+ rm -rf ../yetus_PreCommit-HIVE-Build-16397
+ mkdir ../yetus_PreCommit-HIVE-Build-16397
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16397
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16397/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: does
not exist in index
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:393: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:417: trailing whitespace.
Map 3
/data/hiveptest/working/scratch/build.patch:441: trailing whitespace.
Reducer 2
/data/hiveptest/working/scratch/build.patch:505: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:523: trailing whitespace.
Map 3
warning: squelched 1 whitespace error
warning: 6 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4467477092565910603.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc4467477092565910603.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc2606360822513530397.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787443#comment-16787443
]
Hive QA commented on HIVE-21264:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961600/HIVE-21264.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15822 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=327)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16396/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16396/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961600 - PreCommit-HIVE-Build
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787389#comment-16787389
]
Hive QA commented on HIVE-21264:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} serde: The patch generated 0 new + 51 unchanged - 2
fixed = 51 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16396/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16396/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787382#comment-16787382
]
Ashutosh Chauhan commented on HIVE-21397:
-
On Hive2 it was reported that bloom filters arent created either but actually
results in an exception:
h4. REPRODUCE STEPS
Install a cluster with ACID enabled
{code}
CREATE TABLE IF NOT EXISTS emp_part_bckt (
empid int,
name string,
designation string,
salary int)
PARTITIONED BY (department String)
clustered by (empid) into 2 buckets
stored as orc
TBLPROPERTIES ('transactional'='true', 'orc.create.index'='true',
'orc.bloom.filter.columns'='empid,name,designation');
hive> INSERT INTO emp_part_bckt PARTITION(department) VALUES (1, 'Hajime',
'Test', 100, 'Support');
{code}
h4. ERROR
{noformat}
Status: Failed
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1503649523886_0030_1_01,
diagnostics=[Task failed, taskId=task_1503649523886_0030_1_01_01,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{},"value":{"_col0":"1","_col1":"Hajime","_col2":"Test","_col3":"100","_col4":"Support"}}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{},"value":{"_col0":"1","_col1":"Hajime","_col2":"Test","_col3":"100","_col4":"Support"}}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:284)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:266)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=0)
{"key":{},"value":{"_col0":"1","_col1":"Hajime","_col2":"Test","_col3":"100","_col4":"Support"}}
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:352)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:274)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ArrayIndexOutOfBoundsException: 4
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:581)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:870)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:977)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:720)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:841)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343)
... 17 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
at
org.apache.hadoop.hive.ql.io.orc.OrcUtils.getColumnSpan(OrcUtils.java:134)
at
org.apache.hadoop.hive.ql.io.orc.OrcUtils.includeColumnsImpl(OrcUtils.java:92)
at
org.apache.hadoop.hive.ql.io.orc.OrcUtils.includeColumns(OrcUtils.java:84)
at
org.apache.hadoop.hive.ql.io.orc.WriterImpl.(WriterImpl.java:217)
at
[jira] [Commented] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[
https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787376#comment-16787376
]
Hive QA commented on HIVE-21286:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961598/HIVE-21286.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15821 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16395/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16395/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16395/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961598 - PreCommit-HIVE-Build
> Hive should support clean-up of previously bootstrapped tables when retry
> from different dump.
> --
>
> Key: HIVE-21286
> URL: https://issues.apache.org/jira/browse/HIVE-21286
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch,
> HIVE-21286.03.patch, HIVE-21286.04.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> If external tables are enabled for replication on an existing repl policy,
> then bootstrapping of external tables are combined with incremental dump.
> If incremental bootstrap load fails with non-retryable error for which user
> will have to manually drop all the external tables before trying with another
> bootstrap dump. For full bootstrap, to retry with different dump, we
> suggested user to drop the DB but in this case they need to manually drop all
> the external tables which is not so user friendly. So, need to handle it in
> Hive side as follows.
> REPL LOAD takes additional config (passed by user in WITH clause) that says,
> drop all the tables which are bootstrapped from previous dump.
> hive.repl.clean.tables.from.bootstrap=
> Hive will use this config only if the current dump is combined bootstrap in
> incremental dump.
> Caution to be taken by user that this config should not be passed if previous
> REPL LOAD (with bootstrap) was successful or any successful incremental
> dump+load happened after "previous_bootstrap_dump_dir".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[
https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787349#comment-16787349
]
Hive QA commented on HIVE-21286:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 18
unchanged - 0 fixed = 28 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16395/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16395/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16395/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive should support clean-up of previously bootstrapped tables when retry
> from different dump.
> --
>
> Key: HIVE-21286
> URL: https://issues.apache.org/jira/browse/HIVE-21286
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch,
> HIVE-21286.03.patch, HIVE-21286.04.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> If external tables are enabled for replication on an existing repl policy,
> then bootstrapping of external tables are combined with incremental dump.
> If incremental bootstrap load fails with non-retryable error for which user
> will have to manually drop all the external tables before trying with
[jira] [Updated] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21404:
Attachment: HIVE-21404.2.patch
Status: Patch Available (was: In Progress)
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch, HIVE-21404.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21404:
Status: In Progress (was: Patch Available)
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787324#comment-16787324
]
Hive QA commented on HIVE-21401:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961547/HIVE-21401.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16394/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16394/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16394/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:34:21.855
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16394/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:34:21.859
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:34:22.904
+ rm -rf ../yetus_PreCommit-HIVE-Build-16394
+ mkdir ../yetus_PreCommit-HIVE-Build-16394
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16394
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16394/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java:
does not exist in index
error:
a/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java:
does not exist in index
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook.java:
does not exist in index
error:
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook1.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperation.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperationContext.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLTask2.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLWork2.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java:
does not exist in index
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787321#comment-16787321
]
Hive QA commented on HIVE-21406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961545/Screen%20Shot%202019-03-07%20at%202.02.10%20PM.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16393/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16393/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:33:01.706
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16393/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:33:01.710
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
1a37d05..bd95a2f master -> origin/master
05e0847..4fd9425 branch-3 -> origin/branch-3
008eda8..7dc47fa branch-3.1 -> origin/branch-3.1
+ git reset --hard HEAD
HEAD is now at 1a37d05 HIVE-20848 : After setting UpdateInputAccessTimeHook
query fail with Table Not Found. (Rajkumar Singh via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at bd95a2f HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-07 23:33:05.185
+ rm -rf ../yetus_PreCommit-HIVE-Build-16393
+ mkdir ../yetus_PreCommit-HIVE-Build-16393
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16393
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16393/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16393
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961545 - PreCommit-HIVE-Build
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787320#comment-16787320
]
Hive QA commented on HIVE-21404:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961536/HIVE-21404.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[2]
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16392/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16392/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16392/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961536 - PreCommit-HIVE-Build
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21398) Columns which has estimated statistics should not be considered as unique keys
[
https://issues.apache.org/jira/browse/HIVE-21398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787273#comment-16787273
]
Ashutosh Chauhan commented on HIVE-21398:
-
+1
> Columns which has estimated statistics should not be considered as unique keys
> --
>
> Key: HIVE-21398
> URL: https://issues.apache.org/jira/browse/HIVE-21398
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21398.01.patch
>
>
> Right now for a column to qualify as a unique column it has to meet the
> criteria:
> {code}
> NDV >= numRows
> {code}
> when numRows is 1 this tends to be true ; but numRows is also 1 in cases when
> we are kinda operate in the blind - don't know how many row there are - more
> generatlly: with estimated column statistics.
> As a sideeffect of qualifying all columns to be unique; after a few joins all
> column combinations became uniqueso for a join between 3 tables which
> have (i,j,k) columns; then it will allocate {{i*j*k}} triplets of "unique
> column triplets".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Patch Available (was: Open)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Open (was: Patch Available)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Attachment: HIVE-21382.2.patch
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21376) Incompatible change in Hive bucket computation
[
https://issues.apache.org/jira/browse/HIVE-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21376:
---
Resolution: Fixed
Fix Version/s: 3.1.2
3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-3, branch-3.1. Thanks [~ashutoshc]
> Incompatible change in Hive bucket computation
> --
>
> Key: HIVE-21376
> URL: https://issues.apache.org/jira/browse/HIVE-21376
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: David Phillips
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.2
>
> Attachments: HIVE-21376.01.patch, HIVE-21376.patch
>
>
> HIVE-20007 seems to have inadvertently changed the bucket hash code
> computation via {{ObjectInspectorUtils.getBucketHashCodeOld()}} for the
> {{DATE}} and {{TIMESTAMP}} data type2.
> {{DATE}} was previously computed using {{DateWritable}}, which uses
> {{daysSinceEpoch}} as the hash code. It is now computed using
> {{DateWritableV2}}, which uses the hash code of {{java.time.LocalDate}}
> (which is not days since epoch).
> {{TIMESTAMP}} was previous computed using {{TimestampWritable}} and now uses
> {{TimestampWritableV2}}. They ostensibly use the same hash code computation,
> but there are two important differences:
> # {{TimestampWritable}} rounds the number of milliseconds into the seconds
> portion of the computation, but {{TimestampWritableV2}} does not.
> # {{TimestampWritable}} gets the epoch time from {{java.sql.Timestamp}},
> which returns it relative to the JVM time zone, not UTC.
> {{TimestampWritableV2}} uses a {{LocalDateTime}} relative to UTC.
> I was unable to get Hive 3.1 running in order to verify if this actually
> causes data to be read or written incorrectly (there may be code above this
> library method which makes things work correctly). However, if my
> understanding is correct, this means Hive 3.1 is both forwards and backwards
> incompatible with bucketed tables using either of these data types. It also
> indicates that Hive needs tests to verify that the hash code does not change
> between releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787279#comment-16787279
]
Hive QA commented on HIVE-21404:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16392/dev-support/hive-personality.sh
|
| git revision | master / ab5d389 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16392/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21385) Allow disabling pushdown of non-splittable computation to JDBC sources
[ https://issues.apache.org/jira/browse/HIVE-21385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787278#comment-16787278 ] Daniel Dai commented on HIVE-21385: --- Discussed offline, I am fine to use a on/off button, with a vague name suggest this is just a workaround to push less computation to jdbc. Maybe "hive.jdbc.pushdown.safe"? > Allow disabling pushdown of non-splittable computation to JDBC sources > -- > > Key: HIVE-21385 > URL: https://issues.apache.org/jira/browse/HIVE-21385 > Project: Hive > Issue Type: Improvement > Components: CBO, StorageHandler >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21385.01.patch, HIVE-21385.01.patch, > HIVE-21385.patch > > > Until pushdown is cost-based decision, we will be able to enable / disable > pushdown of operators that prevent reading results from the JDBC connection > in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787265#comment-16787265
]
Hive QA commented on HIVE-21403:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961531/HIVE-21403.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rebuild]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16391/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16391/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16391/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961531 - PreCommit-HIVE-Build
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as target database is bootstrapped from some other path :
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/c3e5ec9e-d951-48aa-b3f4-9aeaf5e010ea.)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20848) After setting UpdateInputAccessTimeHook query fail with Table Not Found.
[
https://issues.apache.org/jira/browse/HIVE-20848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20848:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Rajkumar!
> After setting UpdateInputAccessTimeHook query fail with Table Not Found.
>
>
> Key: HIVE-20848
> URL: https://issues.apache.org/jira/browse/HIVE-20848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20848.01.patch, HIVE-20848.patch
>
>
> {code}
> select from_unixtime(1540495168);
> set
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
> select from_unixtime(1540495168);
> {code}
> the second select fail with following exception
> {code}
> ERROR ql.Driver: FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found
> _dummy_table)
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1217)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1168)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1155)
> at
> org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:67)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1444)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1294)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20546: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Nishant! > Upgrade to Apache Druid 0.13.0-incubating > - > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, > HIVE-20546.3.patch, HIVE-20546.4.patch, HIVE-20546.5.patch, > HIVE-20546.6.patch, HIVE-20546.7.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[ https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21399: --- Attachment: HIVE-21399.03.patch > Adjust hive.map.aggr.hash.min.reduction statically depending on group by > statistics > --- > > Key: HIVE-21399 > URL: https://issues.apache.org/jira/browse/HIVE-21399 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch, > HIVE-21399.03.patch > > > Currently, the value is set statically from config variable. If stats are > available, we could try to adjust this value at optimization time to favor > turning off hash aggregation earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Attachment: HIVE-21382.2.patch
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Patch Available (was: Open)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Open (was: Patch Available)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21400) Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
[
https://issues.apache.org/jira/browse/HIVE-21400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787261#comment-16787261
]
Prasanth Jayachandran commented on HIVE-21400:
--
lgtm, +1
> Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
> -
>
> Key: HIVE-21400
> URL: https://issues.apache.org/jira/browse/HIVE-21400
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21400.1.patch
>
>
> GC thrash from an unexpected source in ReduceSinkOperator.
> {code}
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.resetWithoutOutput(LazyBinarySerializeWrite.java:136)
> at
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.reset(LazyBinarySerializeWrite.java:132)
> at
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkUniformHashOperator.process(VectorReduceSinkUniformHashOperator.java:180)
> {code}
> GC space is getting thrashed by the
> {code}
> root = new Field(STRUCT);
> {code}
> for every row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17668) Push filter clauses through PTF(Windowing) does not work in some cases
[ https://issues.apache.org/jira/browse/HIVE-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787250#comment-16787250 ] Ashutosh Chauhan commented on HIVE-17668: - Ton of golden files needs updating. > Push filter clauses through PTF(Windowing) does not work in some cases > -- > > Key: HIVE-17668 > URL: https://issues.apache.org/jira/browse/HIVE-17668 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0, 2.2.0, 2.3.0, 3.0.0, 2.4.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-17668.01.patch, HIVE-17668.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21392) Misconfigurations of DataNucleus log in log4j.properties
[ https://issues.apache.org/jira/browse/HIVE-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21392: Status: Patch Available (was: Open) > Misconfigurations of DataNucleus log in log4j.properties > > > Key: HIVE-21392 > URL: https://issues.apache.org/jira/browse/HIVE-21392 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.0.0 >Reporter: Chen Zhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21392.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In the patch of > [HIVE-12020|https://issues.apache.org/jira/browse/HIVE-12020], we changed the > DataNucleus related logging configuration from nine fine-grained loggers with > three coarse-grained loggers (DataNucleus, Datastore and JPOX). As Prasanth > Jayachandran > [explain|https://issues.apache.org/jira/browse/HIVE-12020?focusedCommentId=15025612=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15025612], > these three loggers are the top-level logger in DataNucleus, so that we > don't need to specify other loggers for DataNucleus. However, according to > the > [documents|http://www.datanucleus.org/products/accessplatform/logging.html] > and [source > codes|https://github.com/datanucleus/datanucleus-core/blob/master/src/main/java/org/datanucleus/util/NucleusLogger.java#L108] > of DataNucleus, the top-level logger in DataNucleus is `DataNucleus`. > Therefore, we just need to keep the right one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21376) Incompatible change in Hive bucket computation
[
https://issues.apache.org/jira/browse/HIVE-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787245#comment-16787245
]
Ashutosh Chauhan commented on HIVE-21376:
-
+1
> Incompatible change in Hive bucket computation
> --
>
> Key: HIVE-21376
> URL: https://issues.apache.org/jira/browse/HIVE-21376
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: David Phillips
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21376.01.patch, HIVE-21376.patch
>
>
> HIVE-20007 seems to have inadvertently changed the bucket hash code
> computation via {{ObjectInspectorUtils.getBucketHashCodeOld()}} for the
> {{DATE}} and {{TIMESTAMP}} data type2.
> {{DATE}} was previously computed using {{DateWritable}}, which uses
> {{daysSinceEpoch}} as the hash code. It is now computed using
> {{DateWritableV2}}, which uses the hash code of {{java.time.LocalDate}}
> (which is not days since epoch).
> {{TIMESTAMP}} was previous computed using {{TimestampWritable}} and now uses
> {{TimestampWritableV2}}. They ostensibly use the same hash code computation,
> but there are two important differences:
> # {{TimestampWritable}} rounds the number of milliseconds into the seconds
> portion of the computation, but {{TimestampWritableV2}} does not.
> # {{TimestampWritable}} gets the epoch time from {{java.sql.Timestamp}},
> which returns it relative to the JVM time zone, not UTC.
> {{TimestampWritableV2}} uses a {{LocalDateTime}} relative to UTC.
> I was unable to get Hive 3.1 running in order to verify if this actually
> causes data to be read or written incorrectly (there may be code above this
> library method which makes things work correctly). However, if my
> understanding is correct, this means Hive 3.1 is both forwards and backwards
> incompatible with bucketed tables using either of these data types. It also
> indicates that Hive needs tests to verify that the hash code does not change
> between releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-21397:
-
Assignee: Denys Kuzmenko
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Patch Available (was: Open)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch,
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch,
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch,
> HIVE-16924.21.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Attachment: HIVE-16924.21.patch
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch,
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch,
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch,
> HIVE-16924.21.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Open (was: Patch Available)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch,
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch,
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch,
> HIVE-16924.21.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21403) Incorrect error code returned when retry bootstrap with different dump.
[
https://issues.apache.org/jira/browse/HIVE-21403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787234#comment-16787234
]
Hive QA commented on HIVE-21403:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 17 new + 20 unchanged - 17
fixed = 37 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16391/dev-support/hive-personality.sh
|
| git revision | master / f071fae |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16391/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16391/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Incorrect error code returned when retry bootstrap with different dump.
> ---
>
> Key: HIVE-21403
> URL: https://issues.apache.org/jira/browse/HIVE-21403
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21403.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When retry incremental bootstrap on a table with different bootstrap dump
> throws 4 as error code instead of 20017.
> {code}
> Error while processing statement: FAILED: Execution Error, return code 4
> from org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.
> InvalidOperationException(message:Load path
> hdfs://ctr-e139-1542663976389-61669-01-03.hwx.site:8020/apps/hive/repl/3d704b34-bf1a-40c9-b70c-57319e6462f6
> not valid as
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787195#comment-16787195
]
Hive QA commented on HIVE-16924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961527/HIVE-16924.20.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15804 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=155)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=92)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16390/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16390/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16390/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961527 - PreCommit-HIVE-Build
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch,
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch,
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787192#comment-16787192
]
Hive QA commented on HIVE-16924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 8 new + 639 unchanged - 13
fixed = 647 total (was 652) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
5s{color} | {color:red} root: The patch generated 8 new + 647 unchanged - 13
fixed = 655 total (was 660) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} ql generated 0 new + 2256 unchanged - 2 fixed = 2256
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16390/dev-support/hive-personality.sh
|
| git revision | master / f071fae |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16390/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16390/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16390/yetus/whitespace-eol.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16390/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch,
> HIVE-16924.12.patch,
[jira] [Updated] (HIVE-21388) Constant UDF is not pushed to JDBCStorage Handler
[
https://issues.apache.org/jira/browse/HIVE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21388:
---
Attachment: HIVE-21388.02.patch
> Constant UDF is not pushed to JDBCStorage Handler
> -
>
> Key: HIVE-21388
> URL: https://issues.apache.org/jira/browse/HIVE-21388
> Project: Hive
> Issue Type: Improvement
> Components: CBO, StorageHandler
>Affects Versions: 4.0.0
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21388.01.patch, HIVE-21388.02.patch,
> HIVE-21388.patch
>
>
> A query involve a Hive UDF which produce a constant value does not push to
> JDBC table. Replacing the UDF with a constant push down works. Ideally, Hive
> shall first do constant folding and then push the computation.
> Here is the example:
> {code}
> explain select PRINCIPAL_NAME from sys.TBL_PRIVS where
> PRINCIPAL_NAME=current_user();
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_3]|
> | Output:["_col0"] |
> | Filter Operator [FIL_2] |
> | predicate:(_col5 = 'hrt_qa') |
> | Select Operator [SEL_1]|
> | Output:["_col5"] |
> | TableScan [TS_0] |
> | Output:["principal_name"],properties:{"hive.sql.query":"SELECT
> `tbl_grant_id`, `create_time`, `grant_option`, `grantor`, `grantor_type`,
> `principal_name`, `principal_type`, `tbl_priv`, `tbl_id`, `authorizer`\nFROM
> `TBL_PRIVS`","hive.sql.query.fieldNames":"tbl_grant_id,create_time,grant_option,grantor,grantor_type,principal_name,principal_type,tbl_priv,tbl_id,authorizer","hive.sql.query.fieldTypes":"bigint,int,int,string,string,string,string,string,bigint,string","hive.sql.query.split":"true"}
> |
> ||
> ++
> {code}
> If I replace current_user() with a constant, the predicate is pushed to table
> scan.
> Also, setting annotation deterministic=true and make initialize() return a
> ConstantObjectInspector of GenericUDFCurrentUser does not make a difference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787144#comment-16787144
]
Hive QA commented on HIVE-21402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961525/HIVE-21402.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15819 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16389/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16389/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16389/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961525 - PreCommit-HIVE-Build
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21293) Fix ambiguity in grammar warnings at compilation time (II)
[
https://issues.apache.org/jira/browse/HIVE-21293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21293:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-3. Thanks [~abstractdog]!
> Fix ambiguity in grammar warnings at compilation time (II)
> --
>
> Key: HIVE-21293
> URL: https://issues.apache.org/jira/browse/HIVE-21293
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-21293.01.patch, HIVE-21293.02.patch
>
>
> These are the warnings at compilation time:
> {code}
> warning(200): IdentifiersParser.g:424:5:
> Decision can match input such as "KW_UNKNOWN" using multiple alternatives: 1,
> 10
> As a result, alternative(s) 10 were disabled for that input
> {code}
> This means that multiple parser rules can match certain query text, possibly
> leading to unexpected errors at parsing time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787114#comment-16787114
]
Hive QA commented on HIVE-21402:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16389/dev-support/hive-personality.sh
|
| git revision | master / 8ab6ced |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16389/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[ https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-21408: -- Attachment: HIVE-21408.1.patch > Disable synthetic join predicates for non-equi joins for unintended cases > - > > Key: HIVE-21408 > URL: https://issues.apache.org/jira/browse/HIVE-21408 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-21408.1.patch > > > With support for synthetic join predicates on non-equi joins, it is important > to make sure those predicates are used only for intended purpose. Currently, > DPP and semi join reduction are not supposed to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[ https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-21408: -- Status: Patch Available (was: In Progress) > Disable synthetic join predicates for non-equi joins for unintended cases > - > > Key: HIVE-21408 > URL: https://issues.apache.org/jira/browse/HIVE-21408 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > With support for synthetic join predicates on non-equi joins, it is important > to make sure those predicates are used only for intended purpose. Currently, > DPP and semi join reduction are not supposed to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[ https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21408 started by Deepak Jaiswal. - > Disable synthetic join predicates for non-equi joins for unintended cases > - > > Key: HIVE-21408 > URL: https://issues.apache.org/jira/browse/HIVE-21408 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > With support for synthetic join predicates on non-equi joins, it is important > to make sure those predicates are used only for intended purpose. Currently, > DPP and semi join reduction are not supposed to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21408) Disable synthetic join predicates for non-equi joins for unintended cases
[ https://issues.apache.org/jira/browse/HIVE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-21408: - > Disable synthetic join predicates for non-equi joins for unintended cases > - > > Key: HIVE-21408 > URL: https://issues.apache.org/jira/browse/HIVE-21408 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > With support for synthetic join predicates on non-equi joins, it is important > to make sure those predicates are used only for intended purpose. Currently, > DPP and semi join reduction are not supposed to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21293) Fix ambiguity in grammar warnings at compilation time (II)
[
https://issues.apache.org/jira/browse/HIVE-21293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787102#comment-16787102
]
Jesus Camacho Rodriguez commented on HIVE-21293:
+1
> Fix ambiguity in grammar warnings at compilation time (II)
> --
>
> Key: HIVE-21293
> URL: https://issues.apache.org/jira/browse/HIVE-21293
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21293.01.patch, HIVE-21293.02.patch
>
>
> These are the warnings at compilation time:
> {code}
> warning(200): IdentifiersParser.g:424:5:
> Decision can match input such as "KW_UNKNOWN" using multiple alternatives: 1,
> 10
> As a result, alternative(s) 10 were disabled for that input
> {code}
> This means that multiple parser rules can match certain query text, possibly
> leading to unexpected errors at parsing time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-16976:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.10.patch,
> HIVE-16976.11.patxh, HIVE-16976.2.patch, HIVE-16976.3.patch,
> HIVE-16976.4.patch, HIVE-16976.5.patch, HIVE-16976.6.patch,
> HIVE-16976.7.patch, HIVE-16976.8.patch, HIVE-16976.9.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Reducer 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized, llap
> File Output Operator [FS_36]
> Group By Operator [GBY_35] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_34]
> Group By Operator [GBY_33] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(1)"]
> Select Operator [SEL_32] (rows=9600142089 width=16)
> Filter Operator [FIL_31] (rows=9600142089 width=16)
> predicate:(_col0 > _col1)
> Map Join Operator [MAPJOIN_30] (rows=28800426268 width=16)
> Conds:(Inner),Output:["_col0","_col1"]
> <-Reducer 4 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_28]
> Group By Operator [GBY_27] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 3 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_26]
> Group By Operator [GBY_25] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(d_date_sk)"]
> Select Operator [SEL_24] (rows=652 width=12)
> Output:["d_date_sk"]
> Filter Operator [FIL_23] (rows=652 width=12)
> predicate:(d_year = 2017)
> TableScan [TS_2] (rows=73049 width=12)
>
> tpcds_bin_partitioned_newschema_orc_1@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE,Output:["d_date_sk","d_year"]
> <-Select Operator [SEL_29] (rows=28800426268 width=8)
> Output:["_col0"]
> TableScan [TS_0] (rows=28800426268 width=172)
>
> tpcds_bin_partitioned_newschema_orc_1@store_sales,store_sales,Tbl:COMPLETE,Col:COMPLETE
> {code}
> The SyntheticJoinPredicate is only injected for equi joins, not for < or >
> scalar subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21385) Allow disabling pushdown of non-splittable computation to JDBC sources
[ https://issues.apache.org/jira/browse/HIVE-21385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21385: --- Attachment: HIVE-21385.01.patch > Allow disabling pushdown of non-splittable computation to JDBC sources > -- > > Key: HIVE-21385 > URL: https://issues.apache.org/jira/browse/HIVE-21385 > Project: Hive > Issue Type: Improvement > Components: CBO, StorageHandler >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21385.01.patch, HIVE-21385.01.patch, > HIVE-21385.patch > > > Until pushdown is cost-based decision, we will be able to enable / disable > pushdown of operators that prevent reading results from the JDBC connection > in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21385) Allow disabling pushdown of non-splittable computation to JDBC sources
[ https://issues.apache.org/jira/browse/HIVE-21385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787098#comment-16787098 ] Jesus Camacho Rodriguez commented on HIVE-21385: [~daijy], we do not want to go into that direction making per operator rules configurable. I believe an on/off button is easier to understand and will not create any false expectations, e.g., that anything in that list will actually be pushed. Cc [~ashutoshc] > Allow disabling pushdown of non-splittable computation to JDBC sources > -- > > Key: HIVE-21385 > URL: https://issues.apache.org/jira/browse/HIVE-21385 > Project: Hive > Issue Type: Improvement > Components: CBO, StorageHandler >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21385.01.patch, HIVE-21385.01.patch, > HIVE-21385.patch > > > Until pushdown is cost-based decision, we will be able to enable / disable > pushdown of operators that prevent reading results from the JDBC connection > in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[
https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787066#comment-16787066
]
Hive QA commented on HIVE-21325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961524/HIVE-21325.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961524 - PreCommit-HIVE-Build
> Hive external table replication failed with Permission denied issue.
>
>
> Key: HIVE-21325
> URL: https://issues.apache.org/jira/browse/HIVE-21325
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> During external table replication the file copy is done in parallel to the
> meta data replication. If the file copy task creates the directory with do as
> set to true, it will create the directory with permission set to the user
> running the repl command. In that case the meta data task while creating the
> table may fail as hive user might not have access to the created directory.
> The fix should be
> # While creating directory, if sql based authentication is enabled, then
> disable storage based authentication for hive user.
> # Currently the created directory has the login user access, it should
> retain the source clusters owner, group and permission.
> # For external table replication don't create the directory during create
> table and add partition.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: HIVE-21264.3.patch
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Status: Patch Available (was: Open)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Status: Open (was: Patch Available)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[
https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787025#comment-16787025
]
Hive QA commented on HIVE-21325:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 18 unchanged - 1
fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16388/dev-support/hive-personality.sh
|
| git revision | master / 8ab6ced |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16388/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive external table replication failed with Permission denied issue.
>
>
> Key: HIVE-21325
> URL: https://issues.apache.org/jira/browse/HIVE-21325
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> During external table replication the file copy is done in parallel to the
> meta data replication. If the file copy task creates the directory with do as
> set to true, it will create the directory with permission set to the user
> running the repl command. In that case the meta data task while creating the
> table may
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Attachment: HIVE-21286.04.patch > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787019#comment-16787019
]
David Mollitor edited comment on HIVE-21264 at 3/7/19 5:48 PM:
---
[~gopalv] Thank you so much for the review.
If you prefer viewing changes in GitHub, I added a PR:
https://github.com/apache/hive/pull/561
So, {{CharTypeInfo}} and {{VarcharTypeInfo}} are decedents of both
{{BaseCharTypeInfo}} and {{PrimitiveTypeInfo}}. And with this proposed change,
{{CharTypeInfo}} and {{VarcharTypeInfo}} do not override the {{hashcode}} or
{{equals}} method. The equality check to ensure that they are the same type
occurs in {{PrimitiveTypeInfo}} and the check that they are of the same length
occurs in {{BaseCharTypeInfo}}. There is no need to do this check explicitly
in the child classes.
I have added a unit test to my patch to demonstrate.
was (Author: belugabehr):
[~gopalv] Thank you so much for the review.
If you prefer viewing changes in GitHub, I added a PR:
https://github.com/apache/hive/pull/561
So, {{CharTypeInfo}} and {{VarcharTypeInfo}} are decedents of both
{{BaseCharTypeInfo}} and {{PrimitiveTypeInfo}}. And with this proposed change,
{{CharTypeInfo}} and {{VarcharTypeInfo}} do not override the {{hashcode}} or
{{equals}} method. The equality check to ensure that they are the same type
occurs in {{PrimitiveTypeInfo} and the check that they are of the same length
occurs in {{BaseCharTypeInfo}}. There is no need to do this check explicitly
in the child classes.
I have added a unit test to my patch to demonstrate.
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787019#comment-16787019
]
David Mollitor commented on HIVE-21264:
---
[~gopalv] Thank you so much for the review.
If you prefer viewing changes in GitHub, I added a PR:
https://github.com/apache/hive/pull/561
So, {{CharTypeInfo}} and {{VarcharTypeInfo}} are decedents of both
{{BaseCharTypeInfo}} and {{PrimitiveTypeInfo}}. And with this proposed change,
{{CharTypeInfo}} and {{VarcharTypeInfo}} do not override the {{hashcode}} or
{{equals}} method. The equality check to ensure that they are the same type
occurs in {{PrimitiveTypeInfo} and the check that they are of the same length
occurs in {{BaseCharTypeInfo}}. There is no need to do this check explicitly
in the child classes.
I have added a unit test to my patch to demonstrate.
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21264:
--
Labels: pull-request-available (was: )
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?focusedWorklogId=209681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209681
]
ASF GitHub Bot logged work on HIVE-21264:
-
Author: ASF GitHub Bot
Created on: 07/Mar/19 17:42
Start Date: 07/Mar/19 17:42
Worklog Time Spent: 10m
Work Description: BELUGABEHR commented on pull request #561: HIVE-21264:
Improvements Around CharTypeInfo
URL: https://github.com/apache/hive/pull/561
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209681)
Time Spent: 10m
Remaining Estimate: 0h
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Status: Patch Available (was: Open) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Status: Open (was: Patch Available) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21400) Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
[
https://issues.apache.org/jira/browse/HIVE-21400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786992#comment-16786992
]
Hive QA commented on HIVE-21400:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961513/HIVE-21400.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15819 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16387/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16387/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16387/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961513 - PreCommit-HIVE-Build
> Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
> -
>
> Key: HIVE-21400
> URL: https://issues.apache.org/jira/browse/HIVE-21400
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21400.1.patch
>
>
> GC thrash from an unexpected source in ReduceSinkOperator.
> {code}
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.resetWithoutOutput(LazyBinarySerializeWrite.java:136)
> at
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.reset(LazyBinarySerializeWrite.java:132)
> at
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkUniformHashOperator.process(VectorReduceSinkUniformHashOperator.java:180)
> {code}
> GC space is getting thrashed by the
> {code}
> root = new Field(STRUCT);
> {code}
> for every row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21337) HMS Metadata migration from Postgres/Derby to other DBs fail
[ https://issues.apache.org/jira/browse/HIVE-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21337: - Resolution: Fixed Fix Version/s: 4.0.0 Target Version/s: 4.0.0 Status: Resolved (was: Patch Available) Fix has been committed to master. Thanks for the review [~ychena]. I do not feel this needs to backported to branch-3 at this point. > HMS Metadata migration from Postgres/Derby to other DBs fail > > > Key: HIVE-21337 > URL: https://issues.apache.org/jira/browse/HIVE-21337 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-21337.2.patch, HIVE-21337.patch > > > Customer recently was migrating from Postgres to Oracle for HMS metastore. > During import of the [exported] data from HMS metastore from postgres, > failures are seen as the COLUMNS_V2.COMMENT is 4000 bytes long whereas oracle > and other schemas define it to be 256 bytes. > This inconsistency in the schema makes the migration cumbersome and manual. > This jira makes this column consistent in length across all databases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21400) Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
[
https://issues.apache.org/jira/browse/HIVE-21400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786932#comment-16786932
]
Hive QA commented on HIVE-21400:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16387/dev-support/hive-personality.sh
|
| git revision | master / 84f766e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16387/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
> -
>
> Key: HIVE-21400
> URL: https://issues.apache.org/jira/browse/HIVE-21400
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21400.1.patch
>
>
> GC thrash from an unexpected source in ReduceSinkOperator.
> {code}
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.resetWithoutOutput(LazyBinarySerializeWrite.java:136)
> at
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.reset(LazyBinarySerializeWrite.java:132)
> at
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkUniformHashOperator.process(VectorReduceSinkUniformHashOperator.java:180)
> {code}
> GC space is getting thrashed by the
> {code}
> root = new Field(STRUCT);
> {code}
> for every row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21377) Using Oracle as HMS DB with DirectSQL
[
https://issues.apache.org/jira/browse/HIVE-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786908#comment-16786908
]
Hive QA commented on HIVE-21377:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961510/HIVE-21377.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15819 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16386/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16386/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16386/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961510 - PreCommit-HIVE-Build
> Using Oracle as HMS DB with DirectSQL
> -
>
> Key: HIVE-21377
> URL: https://issues.apache.org/jira/browse/HIVE-21377
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Bo
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21377.01.patch, HIVE-21377.patch
>
>
> When we use the Oracle as HMS DB, we saw this kind of contents in the HMS log
> accordingly:
> {code:java}
> 2019-02-02 T08:23:57,102 WARN [Thread-12]: metastore.ObjectStore
> (ObjectStore.java:handleDirectSqlError(3741)) - Falling back to ORM path due
> to direct SQL failure (this is not an error): Cannot extract boolean from
> column value 0 at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlBoolean(MetaStoreDirectSql.java:1031)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:728)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:109)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:471)
> at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:462)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$8.getSqlResult(ObjectStore.java:3392)
> {code}
> In Hive, we handle the Postgres, MySQL and Derby for the extractSqlBoolean.
> But Oracle return the 0 or 1 for Boolean. So we need to modify the
> MetastoreDirectSqlUtils.java - [1]
> So, could add this snip in this code?
> {code:java}
> static Boolean extractSqlBoolean(Object value) throws MetaException {
> if (value == null) {
> return null;
> }
> if (value instanceof Boolean) {
> return (Boolean)value;
> }
> if (value instanceof Number) { // add
> try {
> return BooleanUtils.toBooleanObject((BigDecimal) value, 1, 0, null);
> } catch(IllegalArugmentExeception iae){
> // NOOP
> }
> if (value instanceof String) {
> try {
> return BooleanUtils.toBooleanObject((String) value, "Y", "N", null);
> } catch (IllegalArgumentException iae) {
> // NOOP
> }
> }
> throw new MetaException("Cannot extract boolean from column value " +
> value);
> }
> {code}
> [1] -
> https://github.com/apache/hive/blob/f51f108b761f0c88647f48f30447dae12b308f31/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L501-L527
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
