[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855388#comment-16855388
]
Hive QA commented on HIVE-16690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970752/HIVE-16690.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16076 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=192)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17407/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17407/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17407/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970752 - PreCommit-HIVE-Build
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21821) Backport HIVE-21739 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-21821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Shah updated HIVE-21821: --- Attachment: HIVE-21821.branch-3.1.1.patch Status: Patch Available (was: Open) +Thanks, [~alangates] for review. + ++The CTLG_ID being inserted was 1 in test and hence clashing with the default CTLG. Corrected the test with the new patch. > Backport HIVE-21739 to branch-3.1 > - > > Key: HIVE-21821 > URL: https://issues.apache.org/jira/browse/HIVE-21821 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-21821.branch-3.1.1.patch, > HIVE-21821.branch-3.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21821) Backport HIVE-21739 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-21821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855355#comment-16855355 ] Aditya Shah edited comment on HIVE-21821 at 6/4/19 6:01 AM: Thanks, [~alangates] for review. The CTLG_ID being inserted was 1 in test and hence clashing with the default CTLG. Corrected the test with the new patch. was (Author: aditya-shah): +Thanks, [~alangates] for review. + ++The CTLG_ID being inserted was 1 in test and hence clashing with the default CTLG. Corrected the test with the new patch. > Backport HIVE-21739 to branch-3.1 > - > > Key: HIVE-21821 > URL: https://issues.apache.org/jira/browse/HIVE-21821 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-21821.branch-3.1.1.patch, > HIVE-21821.branch-3.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855351#comment-16855351
]
Hive QA commented on HIVE-16690:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
47s{color} | {color:blue} ql in master has 2261 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 3 new + 36 unchanged - 0 fixed
= 39 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17407/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17407/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17407/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855341#comment-16855341 ] Daniel Dai commented on HIVE-21829: --- +1 pending test. > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 3.1.2 > > Attachments: HIVE-21829.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-21829: --- Attachment: HIVE-21829.1.patch > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 3.1.2 > > Attachments: HIVE-21829.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-21829: --- Status: Patch Available (was: Open) > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 3.1.2 > > Time Spent: 10m > Remaining Estimate: 0h > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?focusedWorklogId=253543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253543 ] ASF GitHub Bot logged work on HIVE-21829: - Author: ASF GitHub Bot Created on: 04/Jun/19 05:31 Start Date: 04/Jun/19 05:31 Worklog Time Spent: 10m Work Description: rameeshm commented on pull request #658: HIVE-21829:HiveMetaStore authorization issue with AlterTable and Drop… URL: https://github.com/apache/hive/pull/658 This is to fix a bug in HIVE-21753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253543) Time Spent: 10m Remaining Estimate: 0h > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 3.1.2 > > Time Spent: 10m > Remaining Estimate: 0h > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21829: -- Labels: pull-request-available (was: ) > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 3.1.2 > > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855327#comment-16855327
]
Gopal V commented on HIVE-21817:
Do you have a FIL following a TS?
The simplest pattern looks like
{code}
select count(1) from calls c where year_partition=2018 and date_time_date
between DATE'2018-01-01' and DATE'2018-02-01';
{code}
My benchmark has 86 thousand queries, not every query is hitting this - so I'll
try to narrow down the queries which hit this, but the specific call is coming
from the PCR which detects a TS+FIL
I have a generic call-trace, which goes backwards from
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrOpProcFactory.java#L110
+
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java#L124
!pcr-cache-miss.png!
> CBO: PartitionPruner cache-miss from Calcite planner
>
>
> Key: HIVE-21817
> URL: https://issues.apache.org/jira/browse/HIVE-21817
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-ppr.png, pcr-cache-miss.png
>
>
> !calcite-ppr.png!
> {code}
> public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
> HiveConf conf, String alias, Map
> prunedPartitionsMap)
> throws SemanticException {
> vs
> public PrunedPartitionList getPrunedPartitions(String alias,
> TableScanOperator ts)
> throws SemanticException {
> PrunedPartitionList partsList = opToPartList.get(ts);
> {code}
> Only the 2nd one hits the PPR cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-21817:
---
Attachment: pcr-cache-miss.png
> CBO: PartitionPruner cache-miss from Calcite planner
>
>
> Key: HIVE-21817
> URL: https://issues.apache.org/jira/browse/HIVE-21817
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-ppr.png, pcr-cache-miss.png
>
>
> !calcite-ppr.png!
> {code}
> public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
> HiveConf conf, String alias, Map
> prunedPartitionsMap)
> throws SemanticException {
> vs
> public PrunedPartitionList getPrunedPartitions(String alias,
> TableScanOperator ts)
> throws SemanticException {
> PrunedPartitionList partsList = opToPartList.get(ts);
> {code}
> Only the 2nd one hits the PPR cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21826) Backport HIVE-21786 to branch-3
[
https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855316#comment-16855316
]
Hive QA commented on HIVE-21826:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970751/HIVE-21826.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17406/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17406/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17406/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-06-04 05:11:21.861
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-17406/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-06-04 05:11:21.866
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f62379b HIVE-21784: Insert overwrite on an acid (not mm) table
is ineffective if the input is empty (Ivan Suller via Laszlo Bodor)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f62379b HIVE-21784: Insert overwrite on an acid (not mm) table
is ineffective if the input is empty (Ivan Suller via Laszlo Bodor)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-06-04 05:11:22.893
+ rm -rf ../yetus_PreCommit-HIVE-Build-17406
+ mkdir ../yetus_PreCommit-HIVE-Build-17406
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-17406
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17406/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: hcatalog/server-extensions/pom.xml:47
Falling back to three-way merge...
Applied patch to 'hcatalog/server-extensions/pom.xml' cleanly.
error: patch failed: pom.xml:25
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
error: patch failed: testutils/ptest2/pom.xml:38
Falling back to three-way merge...
Applied patch to 'testutils/ptest2/pom.xml' cleanly.
Going to apply patch with: git apply -p0
error: patch failed: hcatalog/server-extensions/pom.xml:47
Falling back to three-way merge...
Applied patch to 'hcatalog/server-extensions/pom.xml' cleanly.
error: patch failed: pom.xml:25
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
error: patch failed: testutils/ptest2/pom.xml:38
Falling back to three-way merge...
Applied patch to 'testutils/ptest2/pom.xml' cleanly.
U pom.xml
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-17406
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970751 - PreCommit-HIVE-Build
> Backport HIVE-21786 to branch-3
> ---
>
> Key: HIVE-21826
> URL: https://issues.apache.org/jira/browse/HIVE-21826
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 3.2.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-21826.patch
>
>
> Missed branch-3 in the original fix. Need to apply the patch here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21753) Update HiveMetastore authorization to enable use of HiveAuthorizer implementation
[ https://issues.apache.org/jira/browse/HIVE-21753?focusedWorklogId=253541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253541 ] ASF GitHub Bot logged work on HIVE-21753: - Author: ASF GitHub Bot Created on: 04/Jun/19 05:12 Start Date: 04/Jun/19 05:12 Worklog Time Spent: 10m Work Description: rameeshm commented on pull request #653: HIVE-21753: Update HiveMetastore authorization to enable use of HiveA… URL: https://github.com/apache/hive/pull/653 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253541) Time Spent: 5h 50m (was: 5h 40m) > Update HiveMetastore authorization to enable use of HiveAuthorizer > implementation > - > > Key: HIVE-21753 > URL: https://issues.apache.org/jira/browse/HIVE-21753 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21753.1.patch, HIVE-21753.2.patch, > HIVE-21753.3.patch, HIVE-21753.4.patch, HIVE-21753.5.patch, > HIVE-21753.6.patch, HIVE-21753.7.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Currently HMS supports authorization using StorageBasedAuthorizationProvider > which relies on permissions at filesystem – like HDFS. Hive supports a > pluggable authorization interface, and multiple authorizer implementations > (like SQLStd, Ranger, Sentry) are available to authorizer access in Hive. > Extending HiveMetastore to use the same authorization interface as Hive will > enable use of pluggable authorization implementations; and will result in > consistent authorization across Hive, HMS and other services that use HMS > (like Spark). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21829) HiveMetaStore authorization issue with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-21829: --- Summary: HiveMetaStore authorization issue with AlterTable and DropTable events (was: HiveMetaStore authorization with AlterTable and DropTable events) > HiveMetaStore authorization issue with AlterTable and DropTable events > -- > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Fix For: 3.1.2 > > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[
https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855314#comment-16855314
]
Hive QA commented on HIVE-21827:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970750/HIVE-21827.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16076 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17405/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17405/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17405/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970750 - PreCommit-HIVE-Build
> Multiple calls in SemanticAnalyzer do not go through getTableObjectByName
> method
>
>
> Key: HIVE-21827
> URL: https://issues.apache.org/jira/browse/HIVE-21827
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21827.patch
>
>
> This prevents the table object from being cached in the analyzer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21829) HiveMetaStore authorization with AlterTable and DropTable events
[ https://issues.apache.org/jira/browse/HIVE-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani reassigned HIVE-21829: -- Assignee: Ramesh Mani > HiveMetaStore authorization with AlterTable and DropTable events > > > Key: HIVE-21829 > URL: https://issues.apache.org/jira/browse/HIVE-21829 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > Fix For: 3.1.2 > > > With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer > interface to authorizer metastore events. > This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and > DropTable events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855301#comment-16855301 ] Sam An commented on HIVE-21787: --- previous ptest was finally clean. I am submitting one for logging message for eviction, and cache size. > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.7.patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21799) NullPointerException in DynamicPartitionPruningOptimization, when join key is on aggregation column
[
https://issues.apache.org/jira/browse/HIVE-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855299#comment-16855299
]
Vineet Garg commented on HIVE-21799:
The existing logic to get to {{ExprNode}} for the given column name seems wrong.
{code:java}
ColumnInfo columnInfo =
parentOfRS.getSchema().getColumnInfo(internalColName);
{code}
Above gets to the column info of {{parentOfRS}}, assuming {{parentOfRS}} is
outputting a column named {{internalColName}}
{code:java}
ExprNodeDesc exprNode = null;
if ( parentOfRS.getColumnExprMap() != null) {
exprNode = parentOfRS.getColumnExprMap().get(internalColName).clone();
} else {
exprNode = new ExprNodeColumnDesc(columnInfo);
}
{code}
But this logic is looking for the same column {{internalColName}} in
{{columnExprMap}} which is a mapping of {{parentOfRS's}} input column name to
whatever corresponding expression {{parentOfRS}} will emit. This will work only
if {{parentRS}} do not change the input column and output it as it is.
Assuming that {{internalColName}} refers to the column coming out of
{{parentOfRS}} then this should just be
{code:java}
exprNode = new ExprNodeColumnDesc(columnInfo);
{code}
I believe this change will also fix the issue here. In fact it should go ahead
and create semi join instead of returning.
> NullPointerException in DynamicPartitionPruningOptimization, when join key is
> on aggregation column
> ---
>
> Key: HIVE-21799
> URL: https://issues.apache.org/jira/browse/HIVE-21799
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21799.1.patch, HIVE-21799.2.patch,
> HIVE-21799.3.patch
>
>
> Following table/query results in NPE:
> {noformat}
> create table tez_no_dynpart_hashjoin_on_agg(id int, outcome string, eventid
> int) stored as orc;
> explain select a.id, b.outcome from (select id, max(eventid) as event_id_max
> from tez_no_dynpart_hashjoin_on_agg group by id) a
> LEFT OUTER JOIN tez_no_dynpart_hashjoin_on_agg b
> on a.event_id_max = b.eventid;
> {noformat}
> Stack trace:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan(DynamicPartitionPruningOptimization.java:608)
> at
> org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(DynamicPartitionPruningOptimization.java:239)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.runDynamicPartitionPruning(TezCompiler.java:584)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:165)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:159)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12562)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:370)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:671)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1905)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1852)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1847)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:219)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:340)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:676)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:647)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.run
[jira] [Commented] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[
https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855297#comment-16855297
]
Hive QA commented on HIVE-21827:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
48s{color} | {color:blue} ql in master has 2261 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} ql: The patch generated 0 new + 464 unchanged - 1
fixed = 464 total (was 465) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17405/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17405/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Multiple calls in SemanticAnalyzer do not go through getTableObjectByName
> method
>
>
> Key: HIVE-21827
> URL: https://issues.apache.org/jira/browse/HIVE-21827
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21827.patch
>
>
> This prevents the table object from being cached in the analyzer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855276#comment-16855276
]
Hive QA commented on HIVE-21742:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970749/HIVE-21742.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16076 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17404/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17404/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17404/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970749 - PreCommit-HIVE-Build
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17930) ReplChangeManager should use FileSystem object from current thread
[ https://issues.apache.org/jira/browse/HIVE-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17930: Fix Version/s: (was: 4.0.0) 3.0.0 3.1.0 > ReplChangeManager should use FileSystem object from current thread > -- > > Key: HIVE-17930 > URL: https://issues.apache.org/jira/browse/HIVE-17930 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-17930.1.patch > > > ReplChangeManager is a singleton and it has member FileSystem object that is > created during initialization and then is re-used across recycle method calls. > With doAs=true mode, this doesn't work well as the FileSystem object would > have been created using a user different from the current user. > This is also leading to errors with doAs=false mode, with long running HS2 > instances, as it is failing to renew the kerberos tickets (reason for this > effect is unclear). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17930) ReplChangeManager should use FileSystem object from current thread
[ https://issues.apache.org/jira/browse/HIVE-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17930: Resolution: Duplicate Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Already fixed as part of HIVE-19248. > ReplChangeManager should use FileSystem object from current thread > -- > > Key: HIVE-17930 > URL: https://issues.apache.org/jira/browse/HIVE-17930 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17930.1.patch > > > ReplChangeManager is a singleton and it has member FileSystem object that is > created during initialization and then is re-used across recycle method calls. > With doAs=true mode, this doesn't work well as the FileSystem object would > have been created using a user different from the current user. > This is also leading to errors with doAs=false mode, with long running HS2 > instances, as it is failing to renew the kerberos tickets (reason for this > effect is unclear). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855253#comment-16855253
]
Hive QA commented on HIVE-21742:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
55s{color} | {color:blue} ql in master has 2261 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 1 new + 112 unchanged - 0
fixed = 113 total (was 112) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m
25s{color} | {color:red} ql generated 1 new + 2261 unchanged - 0 fixed = 2262
total (was 2261) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Check for oddness that won't work for negative numbers in
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(ASTNode,
boolean, ArrayList, TypeCheckCtx) At TypeCheckProcFactory.java:work for
negative numbers in
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(ASTNode,
boolean, ArrayList, TypeCheckCtx) At TypeCheckProcFactory.java:[line 1299] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17404/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17404/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17404/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17404/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
[jira] [Updated] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic
[
https://issues.apache.org/jira/browse/HIVE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21818:
---
Attachment: HIVE-21818.02.patch
> CBO: Copying TableRelOptHiveTable has metastore traffic
> ---
>
> Key: HIVE-21818
> URL: https://issues.apache.org/jira/browse/HIVE-21818
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21818.01.patch, HIVE-21818.02.patch,
> HIVE-21818.patch
>
>
> While the optimizer is running, whenever it makes a copy of the TableScan to
> perform potential rewrites, there is Hive metastore traffic from inside CBO
> optimizer.
> {code}
> public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory
> typeFactory, List qualifiedTblName,
> RelDataType rowType, Table hiveTblMetadata, List
> hiveNonPartitionCols,
> List hivePartitionCols, List
> hiveVirtualCols, HiveConf hconf,
> Map partitionCache, Map ColumnStatsList> colStatsCache,
> AtomicInteger noColsMissingStats) {
> Pair, List> constraintKeys =
> generateKeys();
> this.keys = constraintKeys.left;
> this.nonNullablekeys = constraintKeys.right;
> this.referentialConstraints = generateReferentialConstraints();
> }
> {code}
> This is triggered from the rules as the partition pruner
> {code}
> protected void perform(RelOptRuleCall call, Filter filter,
> HiveTableScan tScan) {
> // Original table
> RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();
> // Copy original table scan and table
> HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
> RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
> ...
> if (StringUtils.equals(hiveTableCopy.getPartitionListKey(),
> hiveTable.getPartitionListKey())) {
> // Nothing changed, we do not need to produce a new expression
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic
[
https://issues.apache.org/jira/browse/HIVE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855186#comment-16855186
]
Hive QA commented on HIVE-21818:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
20s{color} | {color:blue} ql in master has 2261 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 6 new + 676 unchanged - 6
fixed = 682 total (was 682) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17403/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17403/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17403/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO: Copying TableRelOptHiveTable has metastore traffic
> ---
>
> Key: HIVE-21818
> URL: https://issues.apache.org/jira/browse/HIVE-21818
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21818.01.patch, HIVE-21818.patch
>
>
> While the optimizer is running, whenever it makes a copy of the TableScan to
> perform potential rewrites, there is Hive metastore traffic from inside CBO
> optimizer.
> {code}
> public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory
> typeFactory, List qualifiedTblName,
> RelDataType rowType, Table hiveTblMetadata, List
> hiveNonPartitionCols,
> List hivePartitionCols, List
> hiveVirtualCols, HiveConf hconf,
> Map partitionCache, Map ColumnStatsList> colStatsCache,
> AtomicInteger noColsMissingStats) {
> Pair, List> constraintKeys =
> generateKeys();
> this.keys = constraintKeys.left;
> this.nonNullablekeys = constraintKeys.right;
> this.referentialConstraints = generateReferentialConstraints();
> }
> {code}
> This is triggered from the rules as the partition pruner
> {code}
> protected void perform(RelOptRuleCall call, Filter filter,
> HiveTableScan tScan) {
> // Original table
> RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();
> // Copy original table
[jira] [Commented] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855172#comment-16855172
]
Hive QA commented on HIVE-21816:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970728/HIVE-21816.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16041 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17402/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17402/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970728 - PreCommit-HIVE-Build
> HMS Translation: Refactor tests to work with ACID tables.
> -
>
> Key: HIVE-21816
> URL: https://issues.apache.org/jira/browse/HIVE-21816
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21816.patch, HIVE-21816.patch, HIVE-21816.patch
>
>
> 1) TestHiveMetaStore unit tests does not work for full ACID tables as the
> TransactionalValidationListener enforces that this table use AcidIO. The Orc
> IO files are only included in the hive-exec jars that are not used by tests
> under standalone-metastore module. Even adding a test-scoped dependency on
> hive-exec did not work. I had to relocate these tests into itests.
> 2) Implementation of logic that allows skipping of translation via the use of
> "MANAGERAWMETADATA" capability.
> 3) Fixed some test bugs as the test was not failing originally when the
> createTable failed because of the issue in #1. As a result, about 3 tests
> never ran fully and never failed. The tests now fail if there are issues.
> 4) Refactoring of the code in the DefaultTransformer to make static lists of
> capabilities. The return capabilities now is dependent on the table
> capabilities, the processor capabilities and the accessType assigned to the
> table.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20415) Hive1: Tez Session failed to return if background thread is interrupted
[
https://issues.apache.org/jira/browse/HIVE-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855170#comment-16855170
]
Ashutosh Chauhan commented on HIVE-20415:
-
+1
> Hive1: Tez Session failed to return if background thread is interrupted
> ---
>
> Key: HIVE-20415
> URL: https://issues.apache.org/jira/browse/HIVE-20415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-20415.1-branch-1.patch, HIVE-20415.2-branch-1.patch
>
>
> user canceled the query which interrupts the background thread, because of
> this interrupt background thread fail to put the session back to the pool.
> {code}
> 2018-08-14 15:55:27,581 ERROR exec.Task (TezTask.java:execute(226)) - Failed
> to execute tez graph.
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.returnSession(TezSessionPoolManager.java:176)
> {code}
> we need a similar fix as HIVE-15731
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21805) HiveServer2: Use the fast ShutdownHookManager APIs
[ https://issues.apache.org/jira/browse/HIVE-21805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855168#comment-16855168 ] Ashutosh Chauhan commented on HIVE-21805: - +1 > HiveServer2: Use the fast ShutdownHookManager APIs > -- > > Key: HIVE-21805 > URL: https://issues.apache.org/jira/browse/HIVE-21805 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: regression > Attachments: HIVE-21805.1.patch, > shutdownhookmanager-configuration.png, txnmanager-shutdownhook-2x.png > > > Hadoop ShutDownHookManager calls "new Configuration()" inside if the > parameters are not provided as args. > This unzips jars & looks for the .xml files in the entire classpath. > !shutdownhookmanager-configuration.png! > + > !txnmanager-shutdownhook-2x.png! > Hive has its own impl of ShutDownHookManager, which has also history from the > hadoo one (added to code instead of shims). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855165#comment-16855165
]
Vineet Garg commented on HIVE-21817:
[~gopalv] I am running simple query from {{partition_pruning}} and I see
{{public static PrunedPartitionList prune}} hitting the cache. Both of these
are using same cache. Can you provide an example query where you observe cache
miss?
> CBO: PartitionPruner cache-miss from Calcite planner
>
>
> Key: HIVE-21817
> URL: https://issues.apache.org/jira/browse/HIVE-21817
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-ppr.png
>
>
> !calcite-ppr.png!
> {code}
> public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
> HiveConf conf, String alias, Map
> prunedPartitionsMap)
> throws SemanticException {
> vs
> public PrunedPartitionList getPrunedPartitions(String alias,
> TableScanOperator ts)
> throws SemanticException {
> PrunedPartitionList partsList = opToPartList.get(ts);
> {code}
> Only the 2nd one hits the PPR cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855165#comment-16855165
]
Vineet Garg edited comment on HIVE-21817 at 6/4/19 12:03 AM:
-
[~gopalv] I am running simple query from {{partition_pruning.q}} and I see
{{public static PrunedPartitionList prune}} hitting the cache. Both of these
are using same cache. Can you provide an example query where you observe cache
miss?
was (Author: vgarg):
[~gopalv] I am running simple query from {{partition_pruning}} and I see
{{public static PrunedPartitionList prune}} hitting the cache. Both of these
are using same cache. Can you provide an example query where you observe cache
miss?
> CBO: PartitionPruner cache-miss from Calcite planner
>
>
> Key: HIVE-21817
> URL: https://issues.apache.org/jira/browse/HIVE-21817
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-ppr.png
>
>
> !calcite-ppr.png!
> {code}
> public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
> HiveConf conf, String alias, Map
> prunedPartitionsMap)
> throws SemanticException {
> vs
> public PrunedPartitionList getPrunedPartitions(String alias,
> TableScanOperator ts)
> throws SemanticException {
> PrunedPartitionList partsList = opToPartList.get(ts);
> {code}
> Only the 2nd one hits the PPR cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21799) NullPointerException in DynamicPartitionPruningOptimization, when join key is on aggregation column
[
https://issues.apache.org/jira/browse/HIVE-21799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855157#comment-16855157
]
Jason Dere commented on HIVE-21799:
---
Hey [~vgarg], can you review?
> NullPointerException in DynamicPartitionPruningOptimization, when join key is
> on aggregation column
> ---
>
> Key: HIVE-21799
> URL: https://issues.apache.org/jira/browse/HIVE-21799
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21799.1.patch, HIVE-21799.2.patch,
> HIVE-21799.3.patch
>
>
> Following table/query results in NPE:
> {noformat}
> create table tez_no_dynpart_hashjoin_on_agg(id int, outcome string, eventid
> int) stored as orc;
> explain select a.id, b.outcome from (select id, max(eventid) as event_id_max
> from tez_no_dynpart_hashjoin_on_agg group by id) a
> LEFT OUTER JOIN tez_no_dynpart_hashjoin_on_agg b
> on a.event_id_max = b.eventid;
> {noformat}
> Stack trace:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan(DynamicPartitionPruningOptimization.java:608)
> at
> org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(DynamicPartitionPruningOptimization.java:239)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.runDynamicPartitionPruning(TezCompiler.java:584)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:165)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:159)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12562)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:370)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:671)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1905)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1852)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1847)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:219)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:340)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:676)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:647)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:182)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:59)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855140#comment-16855140
]
Hive QA commented on HIVE-21816:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
26s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 89 new + 842 unchanged - 153 fixed = 931 total (was 995) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/hive-unit: The patch generated 133 new + 0
unchanged - 0 fixed = 133 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 183 unchanged - 0 fixed = 184 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Redundant nullcheck of txnal, which is known to be non-null in
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(List,
List, String) Redundant null check at MetastoreDefaultTransformer.java:is
known to be non-null in
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(List,
List, String) Redundant null check at MetastoreDefaultTransformer.java:[line
275] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17402/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17402/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17402/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17402/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17402/yetus.txt |
| P
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855126#comment-16855126
]
Hive QA commented on HIVE-21787:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970726/HIVE-21787.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16078 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17401/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17401/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17401/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970726 - PreCommit-HIVE-Build
> Metastore table cache enhancement
> -
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch,
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a
> background thread. This is not a very efficient design.
> In this feature, we try to enhance the cache for Tables with LRU to improve
> cache utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-16690: Status: Patch Available (was: Open) Rebased [~aplusplus]'s patch. > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-16690: Attachment: HIVE-16690.2.patch > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21828) Tez: Use a TezStaticConfiguration inherited from TezConfiguration
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-21828: --- Summary: Tez: Use a TezStaticConfiguration inherited from TezConfiguration (was: Tez: Use a TezStaticConfiguration inherited from TezRuntimeConfiguration) > Tez: Use a TezStaticConfiguration inherited from TezConfiguration > - > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21828) Tez: Use a TezStaticConfiguration inherited from TezRuntimeConfiguration
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-21828: -- Assignee: Gopal V > Tez: Use a TezStaticConfiguration inherited from TezRuntimeConfiguration > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner
[
https://issues.apache.org/jira/browse/HIVE-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855112#comment-16855112
]
Jesus Camacho Rodriguez commented on HIVE-21817:
[~vgarg], would you mind to take this one?
> CBO: PartitionPruner cache-miss from Calcite planner
>
>
> Key: HIVE-21817
> URL: https://issues.apache.org/jira/browse/HIVE-21817
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-ppr.png
>
>
> !calcite-ppr.png!
> {code}
> public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
> HiveConf conf, String alias, Map
> prunedPartitionsMap)
> throws SemanticException {
> vs
> public PrunedPartitionList getPrunedPartitions(String alias,
> TableScanOperator ts)
> throws SemanticException {
> PrunedPartitionList partsList = opToPartList.get(ts);
> {code}
> Only the 2nd one hits the PPR cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855111#comment-16855111 ] Gopal V commented on HIVE-21827: +1 tests pending > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21826) Backport HIVE-21786 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21826: -- Status: Patch Available (was: Open) > Backport HIVE-21786 to branch-3 > --- > > Key: HIVE-21826 > URL: https://issues.apache.org/jira/browse/HIVE-21826 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.2.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-21826.patch > > > Missed branch-3 in the original fix. Need to apply the patch here as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21826) Backport HIVE-21786 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21826: -- Attachment: HIVE-21826.patch > Backport HIVE-21786 to branch-3 > --- > > Key: HIVE-21826 > URL: https://issues.apache.org/jira/browse/HIVE-21826 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.2.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-21826.patch > > > Missed branch-3 in the original fix. Need to apply the patch here as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21827: --- Attachment: HIVE-21827.patch > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21827: --- Status: Patch Available (was: In Progress) > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21827: --- Description: This prevents the table object from being cached in the analyzer. > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21827 started by Jesus Camacho Rodriguez. -- > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21827: -- Assignee: Jesus Camacho Rodriguez > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21827.patch > > > This prevents the table object from being cached in the analyzer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21827) Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
[ https://issues.apache.org/jira/browse/HIVE-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21827: --- Summary: Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method (was: Multiple calls in Semantic) > Multiple calls in SemanticAnalyzer do not go through getTableObjectByName > method > > > Key: HIVE-21827 > URL: https://issues.apache.org/jira/browse/HIVE-21827 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21826) Backport HIVE-21786 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21826: -- Affects Version/s: 3.2.0 > Backport HIVE-21786 to branch-3 > --- > > Key: HIVE-21826 > URL: https://issues.apache.org/jira/browse/HIVE-21826 > Project: Hive > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Missed branch-3 in the original fix. Need to apply the patch here as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21826) Backport HIVE-21786 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-21826: - > Backport HIVE-21786 to branch-3 > --- > > Key: HIVE-21826 > URL: https://issues.apache.org/jira/browse/HIVE-21826 > Project: Hive > Issue Type: Bug >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Missed branch-3 in the original fix. Need to apply the patch here as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855099#comment-16855099
]
Hive QA commented on HIVE-21787:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
27s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 7 new + 44 unchanged - 165 fixed = 51 total (was 209) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 178 unchanged - 5 fixed = 178 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17401/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17401/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17401/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore table cache enhancement
> -
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch,
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a
> background thread. This is not a very efficient design.
> In this feature, we try to enhance the cache for Tables with LRU to improve
> cache utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21826) Backport HIVE-21786 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-21826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21826: -- Component/s: Build Infrastructure > Backport HIVE-21786 to branch-3 > --- > > Key: HIVE-21826 > URL: https://issues.apache.org/jira/browse/HIVE-21826 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 3.2.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > Missed branch-3 in the original fix. Need to apply the patch here as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253400
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290017784
##
File path: ql/src/test/results/clientpositive/infer_join_preds.q.out
##
@@ -1187,7 +1187,7 @@ STAGE PLANS:
predicate: prid is not null (type: boolean)
Statistics: Num rows: 1 Data size: 668 Basic stats: COMPLETE
Column stats: NONE
Select Operator
-expressions: idp_warehouse_id (type: bigint), prid (type:
bigint), concat(CAST( CASE WHEN (prid is null) THEN (1) ELSE (prid) END AS
STRING), ',', CASE WHEN (prtimesheetid is null) THEN (1) ELSE (prtimesheetid)
END, ',', CASE WHEN (prassignmentid is null) THEN (1) ELSE (prassignmentid)
END, ',', CASE WHEN (prchargecodeid is null) THEN (1) ELSE (prchargecodeid)
END, ',', CASE WHEN (prtypecodeid is null) THEN ('') ELSE (CAST( prtypecodeid
AS STRING)) END, ',', CASE WHEN (practsum is null) THEN (1) ELSE (practsum)
END, ',', CASE WHEN (prsequence is null) THEN (1) ELSE (prsequence) END, ',',
CASE WHEN (length(prmodby) is null) THEN ('') ELSE (prmodby) END, ',', CASE
WHEN (prmodtime is null) THEN (TIMESTAMP'2017-12-08 00:00:00') ELSE (prmodtime)
END, ',', CASE WHEN (prrmexported is null) THEN (1) ELSE (prrmexported) END,
',', CASE WHEN (prrmckdel is null) THEN (1) ELSE (prrmckdel) END, ',', CASE
WHEN (slice_status is null) THEN (1) ELSE (slice_status) END, ',', CASE WHEN
(role_id is null) THEN (1) ELSE (role_id) END, ',', CASE WHEN
(length(user_lov1) is null) THEN ('') ELSE (user_lov1) END, ',', CASE WHEN
(length(user_lov2) is null) THEN ('') ELSE (user_lov2) END, ',', CASE WHEN
(incident_id is null) THEN (1) ELSE (incident_id) END, ',', CASE WHEN
(incident_investment_id is null) THEN (1) ELSE (incident_investment_id) END,
',', CASE WHEN (odf_ss_actuals is null) THEN (1) ELSE (odf_ss_actuals) END)
(type: string)
+expressions: idp_warehouse_id (type: bigint), prid (type:
bigint), concat(CAST( CASE WHEN (prid is null) THEN (1L) ELSE (prid) END AS
STRING), ',', CASE WHEN (prtimesheetid is null) THEN (1L) ELSE (prtimesheetid)
END, ',', CASE WHEN (prassignmentid is null) THEN (1L) ELSE (prassignmentid)
END, ',', CASE WHEN (prchargecodeid is null) THEN (1L) ELSE (prchargecodeid)
END, ',', CASE WHEN (prtypecodeid is null) THEN ('') ELSE (CAST( prtypecodeid
AS STRING)) END, ',', CASE WHEN (practsum is null) THEN (CAST( 1 AS
decimal(38,20))) ELSE (practsum) END, ',', CASE WHEN (prsequence is null) THEN
(1L) ELSE (prsequence) END, ',', CASE WHEN (length(prmodby) is null) THEN ('')
ELSE (CAST( prmodby AS STRING)) END, ',', CASE WHEN (prmodtime is null) THEN
(TIMESTAMP'2017-12-08 00:00:00') ELSE (prmodtime) END, ',', CASE WHEN
(prrmexported is null) THEN (1L) ELSE (prrmexported) END, ',', CASE WHEN
(prrmckdel is null) THEN (1L) ELSE (prrmckdel) END, ',', CASE WHEN
(slice_status is null) THEN (1) ELSE (slice_status) END, ',', CASE WHEN
(role_id is null) THEN (1L) ELSE (role_id) END, ',', CASE WHEN
(length(user_lov1) is null) THEN ('') ELSE (CAST( user_lov1 AS STRING)) END,
',', CASE WHEN (length(user_lov2) is null) THEN ('') ELSE (CAST( user_lov2 AS
STRING)) END, ',', CASE WHEN (incident_id is null) THEN (1L) ELSE (incident_id)
END, ',', CASE WHEN (incident_investment_id is null) THEN (1L) ELSE
(incident_investment_id) END, ',', CASE WHEN (odf_ss_actuals is null) THEN (1L)
ELSE (odf_ss_actuals) END) (type: string)
Review comment:
Yes. The type of whole case expression is determined to be `bigint` as a
result `1` is cast to long.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253400)
Time Spent: 2h 40m (was: 2.5h)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253399
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290013915
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
##
@@ -571,7 +571,7 @@ private RexNode handleExplicitCast(ExprNodeGenericFuncDesc
func, List c
* If a CASE has branches with string/int/boolean branch types; there is no
common type.
*/
private List adjustCaseBranchTypes(List nodes, RelDataType
retType) {
-List branchTypes = new ArrayList<>();
Review comment:
Removing it. I'll update the pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253399)
Time Spent: 2.5h (was: 2h 20m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253397
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290052428
##
File path: ql/src/test/results/clientpositive/perf/spark/query80.q.out
##
@@ -782,7 +782,7 @@ STAGE PLANS:
1 Map 33
Statistics: Num rows: 231917759 Data size: 31534108438 Basic
stats: COMPLETE Column stats: NONE
Select Operator
-expressions: _col15 (type: string), _col5 (type:
decimal(7,2)), CASE WHEN (_col9 is not null) THEN (_col9) ELSE (0) END (type:
decimal(12,2)), (_col6 - CASE WHEN (_col10 is not null) THEN (_col10) ELSE (0)
END) (type: decimal(13,2))
+expressions: _col15 (type: string), _col5 (type:
decimal(7,2)), CASE WHEN (_col9 is not null) THEN (_col9) ELSE (CAST( 0 AS
decimal(7,2))) END (type: decimal(7,2)), (_col6 - CASE WHEN (_col10 is not
null) THEN (_col10) ELSE (CAST( 0 AS decimal(7,2))) END) (type: decimal(8,2))
Review comment:
change to 7,2 is due to the `_col9`'s type which is decimal(7,2). The whole
expression's type is determined to be decimal(8,2) for some reason, but this
change is in line with previous logic where even though internal type was
decimal(12,2) whole case expr was decimal(13,2). In short type inference looks
to be more tight now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253397)
Time Spent: 2h 10m (was: 2h)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21742:
---
Attachment: HIVE-21742.4.patch
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21742:
---
Status: Patch Available (was: Open)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21742:
---
Status: Open (was: Patch Available)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253395&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253395
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290021538
##
File path: ql/src/test/results/clientpositive/vectorized_case.q.out
##
@@ -705,13 +705,13 @@ STAGE PLANS:
native: true
vectorizationSchemaColumns:
[0:member:decimal(10,0)/DECIMAL_64, 1:attr:decimal(10,0)/DECIMAL_64,
2:ROW__ID:struct]
Select Operator
- expressions: CASE WHEN ((member = 1)) THEN ((attr + 1)) ELSE (2)
END (type: decimal(11,0))
+ expressions: CASE WHEN ((member = 1)) THEN ((attr + 1)) ELSE
(CAST( 2 AS decimal(11,0))) END (type: decimal(11,0))
outputColumnNames: _col0
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [9]
- selectExpressions: IfExprDecimalColumnColumn(col 6:boolean,
col 10:decimal(11,0)col 8:decimal(11,0))(children:
Decimal64ColEqualDecimal64Scalar(col 0:decimal(10,0)/DECIMAL_64, decimal64Val
1, decimalVal 1) -> 6:boolean, ConvertDecimal64ToDecimal(col
7:decimal(11,0)/DECIMAL_64)(children: Decimal64ColAddDecimal64Scalar(col
1:decimal(10,0)/DECIMAL_64, decimal64Val 1, decimalVal 1) ->
7:decimal(11,0)/DECIMAL_64) -> 10:decimal(11,0), ConstantVectorExpression(val
2) -> 8:decimal(11,0)) -> 9:decimal(11,0)
+ projectedOutputColumnNums: [8]
+ selectExpressions: IfExprDecimal64ColumnDecimal64Scalar(col
6:boolean, col 7:decimal(11,0)/DECIMAL_64, decimal64Val 2, decimalVal
2)(children: Decimal64ColEqualDecimal64Scalar(col 0:decimal(10,0)/DECIMAL_64,
decimal64Val 1, decimalVal 1) -> 6:boolean, Decimal64ColAddDecimal64Scalar(col
1:decimal(10,0)/DECIMAL_64, decimal64Val 1, decimalVal 1) ->
7:decimal(11,0)/DECIMAL_64) -> 8:decimal(11,0)/DECIMAL_64
Review comment:
Yeah I see this change in quite a few tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253395)
Time Spent: 1h 50m (was: 1h 40m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253403
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290015489
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1274,6 +1274,40 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
children);
}
+// introduce cast for expressions which are of different type than
parent WHEN expr's type
Review comment:
WHEN here refers to the whole expression (instead of WHEN branch). I'll
update the comment to make it more clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253403)
Time Spent: 3h (was: 2h 50m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253402&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253402
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290014959
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1274,6 +1274,40 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
children);
}
+// introduce cast for expressions which are of different type than
parent WHEN expr's type
+// this is done so that vectorization is able to vectorize
appropriately
+if (genericUDF instanceof GenericUDFWhen && desc instanceof
ExprNodeGenericFuncDesc) {
+ String castTypeName = desc.getTypeInfo().getTypeName();
+ if (desc.getTypeInfo() instanceof DecimalTypeInfo) {
Review comment:
This special case is to get to appropriate FunctionInfo. For decimals we
need "decimal" instead of "decimal(precision, scale)".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253402)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253404&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253404
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290042782
##
File path: ql/src/test/results/clientpositive/perf/spark/query77.q.out
##
@@ -826,7 +826,7 @@ STAGE PLANS:
outputColumnNames: _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 95833780 Data size: 13030622681 Basic
stats: COMPLETE Column stats: NONE
Select Operator
- expressions: 'web channel' (type: string), _col3 (type:
int), _col4 (type: decimal(17,2)), CASE WHEN (_col1 is not null) THEN (_col1)
ELSE (0) END (type: decimal(17,2)), (_col5 - CASE WHEN (_col2 is not null) THEN
(_col2) ELSE (0) END) (type: decimal(18,2))
+ expressions: 'web channel' (type: string), _col3 (type:
int), _col4 (type: decimal(17,2)), CASE WHEN (_col1 is not null) THEN (_col1)
ELSE (CAST( 0 AS decimal(17,2))) END (type: decimal(17,2)), (_col5 - CASE WHEN
(_col2 is not null) THEN (_col2) ELSE (CAST( 0 AS decimal(17,2))) END) (type:
decimal(18,2))
Review comment:
This is due to calcite planner introducing case expression with mismatched
type. Hive planner introduces cast to fix this and there is no (FULL) constant
propagation afterwards. I'll do a follow-up to investigate and properly fix
this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253404)
Time Spent: 3h 10m (was: 3h)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253398
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290043890
##
File path: ql/src/test/results/clientpositive/llap/subquery_select.q.out
##
@@ -3943,10 +3943,14 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: COMPLETE
-Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
Column stats: COMPLETE
- value expressions: _col0 (type: int)
+Select Operator
Review comment:
I don't see the join type being changed. The change only introduced an
additional SELECT op. Can you point me to it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253398)
Time Spent: 2h 20m (was: 2h 10m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253401&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253401
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290016533
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1274,6 +1274,40 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
children);
}
+// introduce cast for expressions which are of different type than
parent WHEN expr's type
+// this is done so that vectorization is able to vectorize
appropriately
+if (genericUDF instanceof GenericUDFWhen && desc instanceof
ExprNodeGenericFuncDesc) {
+ String castTypeName = desc.getTypeInfo().getTypeName();
+ if (desc.getTypeInfo() instanceof DecimalTypeInfo) {
+castTypeName = serdeConstants.DECIMAL_TYPE_NAME;
+ }
+ FunctionInfo castFun =
FunctionRegistry.getFunctionInfo(castTypeName);
+
+ String whenTypeName = desc.getTypeInfo().getTypeName();
+ List updatedChildren = new ArrayList<>();
+ if (castFun != null) {
+GenericUDF castUDF = castFun.getGenericUDF();
+if (desc.getTypeInfo() instanceof DecimalTypeInfo || desc
+.getTypeInfo() instanceof VarcharTypeInfo || desc
+.getTypeInfo() instanceof CharTypeInfo || desc
+.getTypeInfo() instanceof TimestampLocalTZTypeInfo) {
+ ((SettableUDF) castUDF).setTypeInfo(desc.getTypeInfo());
+}
+for (int i = 0; i < desc.getChildren().size(); i++) {
+ ExprNodeDesc child = children.get(i);
+ ExprNodeDesc newChild = child;
+ if ((i % 2 == 1 || i == desc.getChildren().size() - 1) &&
!whenTypeName
Review comment:
Yes `whenTypeName` here will have scale/precision.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253401)
Time Spent: 2h 50m (was: 2h 40m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21742) Vectorization: CASE result type casting
[
https://issues.apache.org/jira/browse/HIVE-21742?focusedWorklogId=253396&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253396
]
ASF GitHub Bot logged work on HIVE-21742:
-
Author: ASF GitHub Bot
Created on: 03/Jun/19 21:50
Start Date: 03/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #656: HIVE-21742
: Vectorization: CASE result type casting
URL: https://github.com/apache/hive/pull/656#discussion_r290015833
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1274,6 +1274,40 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
children);
}
+// introduce cast for expressions which are of different type than
parent WHEN expr's type
+// this is done so that vectorization is able to vectorize
appropriately
+if (genericUDF instanceof GenericUDFWhen && desc instanceof
ExprNodeGenericFuncDesc) {
+ String castTypeName = desc.getTypeInfo().getTypeName();
+ if (desc.getTypeInfo() instanceof DecimalTypeInfo) {
+castTypeName = serdeConstants.DECIMAL_TYPE_NAME;
+ }
+ FunctionInfo castFun =
FunctionRegistry.getFunctionInfo(castTypeName);
+
+ String whenTypeName = desc.getTypeInfo().getTypeName();
+ List updatedChildren = new ArrayList<>();
+ if (castFun != null) {
+GenericUDF castUDF = castFun.getGenericUDF();
+if (desc.getTypeInfo() instanceof DecimalTypeInfo || desc
+.getTypeInfo() instanceof VarcharTypeInfo || desc
+.getTypeInfo() instanceof CharTypeInfo || desc
+.getTypeInfo() instanceof TimestampLocalTZTypeInfo) {
+ ((SettableUDF) castUDF).setTypeInfo(desc.getTypeInfo());
Review comment:
I am not sure if I understand what do you mean. Can you explain?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253396)
Time Spent: 2h (was: 1h 50m)
> Vectorization: CASE result type casting
> ---
>
> Key: HIVE-21742
> URL: https://issues.apache.org/jira/browse/HIVE-21742
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Vectorization
>Affects Versions: 3.1.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21742.1.patch, HIVE-21742.2.patch,
> HIVE-21742.3.patch, HIVE-21742.4.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code}
> create temporary table foo(q548284 int);
> insert into foo values(1),(2),(3),(4),(5),(6);
> select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2))
> THEN (0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8)
> WHEN ((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit
> 1;
> {code}
> Fails with
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
> {code}
> This gets fixed if the case return of (1) is turned into a (1.0).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21821) Backport HIVE-21739 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-21821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-21821: -- Status: Open (was: Patch Available) > Backport HIVE-21739 to branch-3.1 > - > > Key: HIVE-21821 > URL: https://issues.apache.org/jira/browse/HIVE-21821 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-21821.branch-3.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21821) Backport HIVE-21739 to branch-3.1
[
https://issues.apache.org/jira/browse/HIVE-21821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855083#comment-16855083
]
Alan Gates commented on HIVE-21821:
---
When I run apply this patch and run TestSchemaToolForMetastore test I see:
{{[INFO] Running
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore}}
{{[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
58.965 s <<< FAILURE! - in
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore}}
{{[ERROR]
testValidateLocations(org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore)
Time elapsed: 5.398 s <<< ERROR!}}
{{java.io.IOException: Schema script failed, errorcode OTHER}}
{{ at
org.apache.hadoop.hive.metastore.tools.TestSchemaToolForMetastore.testValidateLocations(TestSchemaToolForMetastore.java:358)}}
> Backport HIVE-21739 to branch-3.1
> -
>
> Key: HIVE-21821
> URL: https://issues.apache.org/jira/browse/HIVE-21821
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-21821.branch-3.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855074#comment-16855074
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970724/HIVE-21637.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16076 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17400/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17400/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17400/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970724 - PreCommit-HIVE-Build
> Synchronized metastore cache
>
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch,
> HIVE-21637.3.patch, HIVE-21637.4.patch, HIVE-21637.5.patch, HIVE-21637.6.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is
> asynchronized and in HMS HA setting, we can only get eventual consistency. In
> this Jira, we try to make it synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic
[
https://issues.apache.org/jira/browse/HIVE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855047#comment-16855047
]
Jesus Camacho Rodriguez commented on HIVE-21818:
[~gopalv], I think I solved that in latest patch? :)
> CBO: Copying TableRelOptHiveTable has metastore traffic
> ---
>
> Key: HIVE-21818
> URL: https://issues.apache.org/jira/browse/HIVE-21818
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21818.01.patch, HIVE-21818.patch
>
>
> While the optimizer is running, whenever it makes a copy of the TableScan to
> perform potential rewrites, there is Hive metastore traffic from inside CBO
> optimizer.
> {code}
> public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory
> typeFactory, List qualifiedTblName,
> RelDataType rowType, Table hiveTblMetadata, List
> hiveNonPartitionCols,
> List hivePartitionCols, List
> hiveVirtualCols, HiveConf hconf,
> Map partitionCache, Map ColumnStatsList> colStatsCache,
> AtomicInteger noColsMissingStats) {
> Pair, List> constraintKeys =
> generateKeys();
> this.keys = constraintKeys.left;
> this.nonNullablekeys = constraintKeys.right;
> this.referentialConstraints = generateReferentialConstraints();
> }
> {code}
> This is triggered from the rules as the partition pruner
> {code}
> protected void perform(RelOptRuleCall call, Filter filter,
> HiveTableScan tScan) {
> // Original table
> RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();
> // Copy original table scan and table
> HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
> RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
> ...
> if (StringUtils.equals(hiveTableCopy.getPartitionListKey(),
> hiveTable.getPartitionListKey())) {
> // Nothing changed, we do not need to produce a new expression
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic
[
https://issues.apache.org/jira/browse/HIVE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855043#comment-16855043
]
Gopal V commented on HIVE-21818:
No longer fetching this in the loop, but still fetching it multiple times.
{code}
+PrimaryKeyInfo pki = db.getReliablePrimaryKeys(
+tabMetaData.getDbName(), tabMetaData.getTableName());
+ForeignKeyInfo fki = db.getReliableForeignKeys(
+tabMetaData.getDbName(), tabMetaData.getTableName());
{code}
is not passed into inferNotNullableColumns() and it fetches it again.
> CBO: Copying TableRelOptHiveTable has metastore traffic
> ---
>
> Key: HIVE-21818
> URL: https://issues.apache.org/jira/browse/HIVE-21818
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21818.01.patch, HIVE-21818.patch
>
>
> While the optimizer is running, whenever it makes a copy of the TableScan to
> perform potential rewrites, there is Hive metastore traffic from inside CBO
> optimizer.
> {code}
> public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory
> typeFactory, List qualifiedTblName,
> RelDataType rowType, Table hiveTblMetadata, List
> hiveNonPartitionCols,
> List hivePartitionCols, List
> hiveVirtualCols, HiveConf hconf,
> Map partitionCache, Map ColumnStatsList> colStatsCache,
> AtomicInteger noColsMissingStats) {
> Pair, List> constraintKeys =
> generateKeys();
> this.keys = constraintKeys.left;
> this.nonNullablekeys = constraintKeys.right;
> this.referentialConstraints = generateReferentialConstraints();
> }
> {code}
> This is triggered from the rules as the partition pruner
> {code}
> protected void perform(RelOptRuleCall call, Filter filter,
> HiveTableScan tScan) {
> // Original table
> RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();
> // Copy original table scan and table
> HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
> RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
> ...
> if (StringUtils.equals(hiveTableCopy.getPartitionListKey(),
> hiveTable.getPartitionListKey())) {
> // Nothing changed, we do not need to produce a new expression
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855022#comment-16855022
]
Hive QA commented on HIVE-21637:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
12s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17400/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17400/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Synchronized metastore cache
>
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch,
> HIVE-21637.3.patch, HIVE-21637.4.patch, HIVE-21637.5.patch, HIVE-21637.6.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is
> asynchronized and in HMS HA setting, we can only get eventual consistency. In
> this Jira, we try to make it synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21527) LLAP: Table property to skip cache
[ https://issues.apache.org/jira/browse/HIVE-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-21527: -- Assignee: Gopal V (was: Prasanth Jayachandran) > LLAP: Table property to skip cache > -- > > Key: HIVE-21527 > URL: https://issues.apache.org/jira/browse/HIVE-21527 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Gopal V >Priority: Major > > Similar to HIVE-21305, there can be text tables with big string columns that > are not cache friendly (often pollutes the cache). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855001#comment-16855001
]
Hive QA commented on HIVE-20801:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954529/HIVE-20801.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16077 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17399/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17399/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17399/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954529 - PreCommit-HIVE-Build
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
> Disabling this lock acquisition would improve the performance of non-ACID
> tables co-existing with a globally configured DbTxnManager implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic
[
https://issues.apache.org/jira/browse/HIVE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21818:
---
Attachment: HIVE-21818.01.patch
> CBO: Copying TableRelOptHiveTable has metastore traffic
> ---
>
> Key: HIVE-21818
> URL: https://issues.apache.org/jira/browse/HIVE-21818
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21818.01.patch, HIVE-21818.patch
>
>
> While the optimizer is running, whenever it makes a copy of the TableScan to
> perform potential rewrites, there is Hive metastore traffic from inside CBO
> optimizer.
> {code}
> public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory
> typeFactory, List qualifiedTblName,
> RelDataType rowType, Table hiveTblMetadata, List
> hiveNonPartitionCols,
> List hivePartitionCols, List
> hiveVirtualCols, HiveConf hconf,
> Map partitionCache, Map ColumnStatsList> colStatsCache,
> AtomicInteger noColsMissingStats) {
> Pair, List> constraintKeys =
> generateKeys();
> this.keys = constraintKeys.left;
> this.nonNullablekeys = constraintKeys.right;
> this.referentialConstraints = generateReferentialConstraints();
> }
> {code}
> This is triggered from the rules as the partition pruner
> {code}
> protected void perform(RelOptRuleCall call, Filter filter,
> HiveTableScan tScan) {
> // Original table
> RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();
> // Copy original table scan and table
> HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
> RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
> ...
> if (StringUtils.equals(hiveTableCopy.getPartitionListKey(),
> hiveTable.getPartitionListKey())) {
> // Nothing changed, we do not need to produce a new expression
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Attachment: HIVE-21816.patch > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Status: Patch Available (was: Open) > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Status: Open (was: Patch Available) The failed test looks unrelated to the change. Cancelling and re-attaching the same patch for a clean run. > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854973#comment-16854973
]
Hive QA commented on HIVE-20801:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
5s{color} | {color:blue} ql in master has 2261 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} ql: The patch generated 6 new + 533 unchanged - 0
fixed = 539 total (was 533) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17399/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17399/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17399/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17399/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency desp
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.6.patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19606) Straggler thread in HS2 for rename directory operation stuck in loop causing performance issue and cluster slowdown
[ https://issues.apache.org/jira/browse/HIVE-19606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-19606: --- Assignee: Laszlo Bodor > Straggler thread in HS2 for rename directory operation stuck in loop causing > performance issue and cluster slowdown > --- > > Key: HIVE-19606 > URL: https://issues.apache.org/jira/browse/HIVE-19606 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Laszlo Bodor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854934#comment-16854934 ] Laszlo Bodor commented on HIVE-20801: - [~ashutoshc]: could you please check what the step should be with this? > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854924#comment-16854924
]
Hive QA commented on HIVE-21816:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970717/HIVE-21816.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16041 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=297)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17398/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17398/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17398/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970717 - PreCommit-HIVE-Build
> HMS Translation: Refactor tests to work with ACID tables.
> -
>
> Key: HIVE-21816
> URL: https://issues.apache.org/jira/browse/HIVE-21816
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21816.patch, HIVE-21816.patch
>
>
> 1) TestHiveMetaStore unit tests does not work for full ACID tables as the
> TransactionalValidationListener enforces that this table use AcidIO. The Orc
> IO files are only included in the hive-exec jars that are not used by tests
> under standalone-metastore module. Even adding a test-scoped dependency on
> hive-exec did not work. I had to relocate these tests into itests.
> 2) Implementation of logic that allows skipping of translation via the use of
> "MANAGERAWMETADATA" capability.
> 3) Fixed some test bugs as the test was not failing originally when the
> createTable failed because of the issue in #1. As a result, about 3 tests
> never ran fully and never failed. The tests now fail if there are issues.
> 4) Refactoring of the code in the DefaultTransformer to make static lists of
> capabilities. The return capabilities now is dependent on the table
> capabilities, the processor capabilities and the accessType assigned to the
> table.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854920#comment-16854920 ] Daniel Dai commented on HIVE-21637: --- HIVE-21637.6.patch is just to understand why HiveMetaStoreClient need a new configuration copy. > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch, > HIVE-21637.3.patch, HIVE-21637.4.patch, HIVE-21637.5.patch, HIVE-21637.6.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21637: -- Attachment: HIVE-21637.6.patch > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch, > HIVE-21637.3.patch, HIVE-21637.4.patch, HIVE-21637.5.patch, HIVE-21637.6.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854886#comment-16854886
]
Hive QA commented on HIVE-21816:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
26s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 89 new + 843 unchanged - 153 fixed = 932 total (was 996) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 133 new + 0
unchanged - 0 fixed = 133 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 183 unchanged - 0 fixed = 184 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Redundant nullcheck of txnal, which is known to be non-null in
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(List,
List, String) Redundant null check at MetastoreDefaultTransformer.java:is
known to be non-null in
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(List,
List, String) Redundant null check at MetastoreDefaultTransformer.java:[line
275] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17398/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17398/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17398/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17398/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17398/yetus.txt |
| P
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854825#comment-16854825
]
Hive QA commented on HIVE-21787:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970697/HIVE-21787.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16078 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
(batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17397/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17397/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970697 - PreCommit-HIVE-Build
> Metastore table cache enhancement
> -
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch,
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a
> background thread. This is not a very efficient design.
> In this feature, we try to enhance the cache for Tables with LRU to improve
> cache utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Status: Patch Available (was: Open) > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Attachment: HIVE-21816.patch > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-21816: - Status: Open (was: Patch Available) > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch, HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21816) HMS Translation: Refactor tests to work with ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854772#comment-16854772 ] Naveen Gangam commented on HIVE-21816: -- The test failures do not seem to be related to the fix. I have been able to execute them all locally successfully. [~daijy] > HMS Translation: Refactor tests to work with ACID tables. > - > > Key: HIVE-21816 > URL: https://issues.apache.org/jira/browse/HIVE-21816 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21816.patch > > > 1) TestHiveMetaStore unit tests does not work for full ACID tables as the > TransactionalValidationListener enforces that this table use AcidIO. The Orc > IO files are only included in the hive-exec jars that are not used by tests > under standalone-metastore module. Even adding a test-scoped dependency on > hive-exec did not work. I had to relocate these tests into itests. > 2) Implementation of logic that allows skipping of translation via the use of > "MANAGERAWMETADATA" capability. > 3) Fixed some test bugs as the test was not failing originally when the > createTable failed because of the issue in #1. As a result, about 3 tests > never ran fully and never failed. The tests now fail if there are issues. > 4) Refactoring of the code in the DefaultTransformer to make static lists of > capabilities. The return capabilities now is dependent on the table > capabilities, the processor capabilities and the accessType assigned to the > table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[
https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854751#comment-16854751
]
Hive QA commented on HIVE-21787:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
25s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 7 new + 44 unchanged - 165 fixed = 51 total (was 209) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 178 unchanged - 5 fixed = 178 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17397/dev-support/hive-personality.sh
|
| git revision | master / f62379b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17397/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17397/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore table cache enhancement
> -
>
> Key: HIVE-21787
> URL: https://issues.apache.org/jira/browse/HIVE-21787
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch,
> HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch
>
>
> Metastore currently uses black/white list to specify patterns of tables to
> load into the cache. Cache is loaded in one shot "prewarm", and updated by a
> background thread. This is not a very efficient design.
> In this feature, we try to enhance the cache for Tables with LRU to improve
> cache utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21800) Tez: Cartesian product reparses HiveConf XML
[
https://issues.apache.org/jira/browse/HIVE-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-21800:
---
Attachment: hive3-with-cbo.svg
> Tez: Cartesian product reparses HiveConf XML
> -
>
> Key: HIVE-21800
> URL: https://issues.apache.org/jira/browse/HIVE-21800
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Priority: Minor
> Attachments: Tez-CartesianProductSlowness.png, hive3-with-cbo.svg
>
>
> {code}
> CartesianProductConfig cpConfig = new
> CartesianProductConfig(crossProductSources);
> edgeManagerDescriptor.setUserPayload(cpConfig.toUserPayload(new
> TezConfiguration(conf)));
> {code}
> !Tez-CartesianProductSlowness.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21800) Tez: Cartesian product reparses HiveConf XML
[
https://issues.apache.org/jira/browse/HIVE-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854741#comment-16854741
]
Gopal V commented on HIVE-21800:
[^hive3-with-cbo.svg] : ignore the Calcite Janino problems and look at the
TezTask.execute()
> Tez: Cartesian product reparses HiveConf XML
> -
>
> Key: HIVE-21800
> URL: https://issues.apache.org/jira/browse/HIVE-21800
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Priority: Minor
> Attachments: Tez-CartesianProductSlowness.png, hive3-with-cbo.svg
>
>
> {code}
> CartesianProductConfig cpConfig = new
> CartesianProductConfig(crossProductSources);
> edgeManagerDescriptor.setUserPayload(cpConfig.toUserPayload(new
> TezConfiguration(conf)));
> {code}
> !Tez-CartesianProductSlowness.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21788) Support replication from haddop-2 (hive 3.0 and beelow) on-prem cluster to hadoop-3 (hive 4 and above) cloud cluster
[
https://issues.apache.org/jira/browse/HIVE-21788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854709#comment-16854709
]
Ashutosh Bapat commented on HIVE-21788:
---
{noformat}
Collection redactedProperties =
- jobConf.getStringCollection(MRJobConfig.MR_JOB_REDACTED_PROPERTIES);
+ jobConf.getStringCollection("mapreduce.job.redacted-properties");
// Hide sensitive configuration values from MR HistoryUI by telling MR to
redact the following list.
- jobConf.set(MRJobConfig.MR_JOB_REDACTED_PROPERTIES,
+ jobConf.set("mapreduce.job.redacted-properties",
StringUtils.join(redactedProperties, COMMA));
}{noformat}
Why do we need those changes? Aren't these constants defined when Hadoop-2 is
used? This comment is
applicable to all the places where this change is repeated.
{noformat}
+ if (conf.get("mapreduce.framework.name") != null
+ && conf.get("mapreduce.framework.name").equals("yarn")) {{noformat}
{noformat}
+ jConf.set("yarn.scheduler.capacity.root.queues", "default");
+ jConf.set("yarn.scheduler.capacity.root.default.capacity", "100");
{noformat}
{noformat}
+ public int getJobTrackerPort() throws UnsupportedOperationException {
+ String address = conf.get("yarn.resourcemanager.address");{noformat}
{noformat}
+
+ if (!isLlap) { // Conf for non-llap
+ conf.setBoolean("hive.llap.io.enabled", false);
+ } else { // Conf for llap
+ conf.set("hive.llap.execution.mode", "only");{noformat}
{noformat}
+ conf.setInt("hive.tez.container.size", 128);{noformat}
Can we use ConfVar or some such static declaration instead of a hard-coded
constant? This comment is
applicable to all the places where we are using hard-coded strings for config.
The problem with
hard-coded configs is that if we change the config in future we won't be able
to catch all the
places where it is used and won't be able to change all such places.
Can you please create PR?
> Support replication from haddop-2 (hive 3.0 and beelow) on-prem cluster to
> hadoop-3 (hive 4 and above) cloud cluster
>
>
> Key: HIVE-21788
> URL: https://issues.apache.org/jira/browse/HIVE-21788
> Project: Hive
> Issue Type: Task
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21788.01.patch
>
>
> In case of replication to cloud both dump and load are executed in the source
> cluster. This push based replication is done to avoid computation at target
> cloud cluster. In case in the source cluster, strict managed table is not set
> to true the tables will be non acid. So during replication to a cluster with
> strict managed table, migration logic same as upgrade tool has to be applied
> on the replicated data. This migration logic is implemented only in hive4.0.
> So it's required that a hive 4.0 instance started at the source cluster. If
> the source cluster has hadoop-2 installation, hive4 has to be built with
> hadoop-2 and necessary changes are required in the pom files and the shim
> files.
> 1. Change the pom.xml files to accept a profile for hadoop-2. If hadoop-2
> profile is set, the hadoop version should be set accordingly to hadoop-2.
> 2. In shim creare a new file for hadoop-2. Based on the profile the
> respective file will be included in the build.
> 3. Changed artifactId hadoop-hdfs-client to hadoop-client as in hadoop-2 the
> jars are stored under hadoop-client folder.
>
>
> Command to enable hadop-2 dependency — mvn clean install package
> -DskipTests -Pdist -pl '!standalone-metastore, !llap-common, !llap-client,
> !llap-ext-client, !llap-tez, !llap-server, !hbase-handler, !service, !hplsql,
> !kryo-registrator' -Phadoop-2.7
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854680#comment-16854680 ] Sam An commented on HIVE-21787: --- putting a renamed patch(same content) to trigger ptest again. Somehow the one uploaded on Friday did not trigger anything. > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.5.patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21822) Expose LlapDaemon metrics through a new API method
[ https://issues.apache.org/jira/browse/HIVE-21822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854668#comment-16854668 ] Peter Vary commented on HIVE-21822: --- [~odraese], [~bslim]: Could you please review? Thanks, Peter > Expose LlapDaemon metrics through a new API method > -- > > Key: HIVE-21822 > URL: https://issues.apache.org/jira/browse/HIVE-21822 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21822.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We should add a new API method to access specific LlapDaemon metrics which > would provide information to TezAMs to make good decisions on load balancing > tasks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21822) Expose LlapDaemon metrics through a new API method
[ https://issues.apache.org/jira/browse/HIVE-21822?focusedWorklogId=253134&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253134 ] ASF GitHub Bot logged work on HIVE-21822: - Author: ASF GitHub Bot Created on: 03/Jun/19 14:44 Start Date: 03/Jun/19 14:44 Worklog Time Spent: 10m Work Description: pvary commented on pull request #657: HIVE-21822: Expose LlapDaemon metrics through a new API method URL: https://github.com/apache/hive/pull/657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253134) Time Spent: 10m Remaining Estimate: 0h > Expose LlapDaemon metrics through a new API method > -- > > Key: HIVE-21822 > URL: https://issues.apache.org/jira/browse/HIVE-21822 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21822.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We should add a new API method to access specific LlapDaemon metrics which > would provide information to TezAMs to make good decisions on load balancing > tasks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21822) Expose LlapDaemon metrics through a new API method
[ https://issues.apache.org/jira/browse/HIVE-21822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21822: -- Labels: pull-request-available (was: ) > Expose LlapDaemon metrics through a new API method > -- > > Key: HIVE-21822 > URL: https://issues.apache.org/jira/browse/HIVE-21822 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21822.patch > > > We should add a new API method to access specific LlapDaemon metrics which > would provide information to TezAMs to make good decisions on load balancing > tasks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21797) Break up DDLTask - extract Storage related operations
[ https://issues.apache.org/jira/browse/HIVE-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854627#comment-16854627 ] Zoltan Haindrich commented on HIVE-21797: - +1 > Break up DDLTask - extract Storage related operations > - > > Key: HIVE-21797 > URL: https://issues.apache.org/jira/browse/HIVE-21797 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21797.01.patch, HIVE-21797.02.patch, > HIVE-21797.03.patch, HIVE-21797.04.patch, HIVE-21797.05.patch, > HIVE-21797.06.patch, HIVE-21797.07.patch, HIVE-21797.08.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #10: extract all the storage related operations from the old DDLTask, > and move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
