[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268315#comment-16268315
]
Hive QA commented on HIVE-17361:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 140 new + 1558 unchanged - 28
fixed = 1698 total (was 1586) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8015/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8015/yetus/patch-asflicense-problems.txt
|
| modules | C: common standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8015/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.23.patch, HIVE-17361.24.patch, HIVE-17361.3.patch,
> HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268261#comment-16268261
]
Hive QA commented on HIVE-18157:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899526/HIVE-18157.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11414 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_insert_into_bucketed_table]
(batchId=25)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8014/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8014/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8014/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899526 - PreCommit-HIVE-Build
> Vectorization : Insert in bucketed table is broken with vectorization
> -
>
> Key: HIVE-18157
> URL: https://issues.apache.org/jira/browse/HIVE-18157
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18157.1.patch
>
>
> create temporary table foo (x int) clustered by (x) into 4 buckets;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 9 00_0
> 4 00_0
> 3 00_0
> 2 00_0
> 1 00_0
> set hive.vectorized.execution.enabled=false;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 4 00_0
> 9 01_0
> 1 01_0
> 2 02_0
> 3 03_0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[
https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268200#comment-16268200
]
Hive QA commented on HIVE-18157:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8014/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8014/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization : Insert in bucketed table is broken with vectorization
> -
>
> Key: HIVE-18157
> URL: https://issues.apache.org/jira/browse/HIVE-18157
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18157.1.patch
>
>
> create temporary table foo (x int) clustered by (x) into 4 buckets;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 9 00_0
> 4 00_0
> 3 00_0
> 2 00_0
> 1 00_0
> set hive.vectorized.execution.enabled=false;
> insert overwrite table foo values(1),(2),(3),(4),(9);
> select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo;
> OK
> 4 00_0
> 9 01_0
> 1 01_0
> 2 02_0
> 3 03_0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18043) Vectorization: Support List type in MapWork
[
https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268183#comment-16268183
]
Hive QA commented on HIVE-18043:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899525/HIVE-18043.005.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11414 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8013/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8013/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8013/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899525 - PreCommit-HIVE-Build
> Vectorization: Support List type in MapWork
> ---
>
> Key: HIVE-18043
> URL: https://issues.apache.org/jira/browse/HIVE-18043
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch,
> HIVE-18043.003.patch, HIVE-18043.004.patch, HIVE-18043.005.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but List
> type is still not support in MapWork. It should be supported to improve the
> performance when vectorization is enable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18043) Vectorization: Support List type in MapWork
[
https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268161#comment-16268161
]
Hive QA commented on HIVE-18043:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 684 unchanged - 0
fixed = 685 total (was 684) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
44s{color} | {color:red} root: The patch generated 1 new + 690 unchanged - 0
fixed = 691 total (was 690) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8013/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8013/yetus/diff-checkstyle-root.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8013/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8013/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Support List type in MapWork
> ---
>
> Key: HIVE-18043
> URL: https://issues.apache.org/jira/browse/HIVE-18043
> Project: Hive
> Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch,
> HIVE-18043.003.patch, HIVE-18043.004.patch, HIVE-18043.005.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but List
> type is still not support in MapWork. It should be supported to improve the
> performance when vectorization is enable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268151#comment-16268151
]
Steve Yeom commented on HIVE-17856:
---
Hey Sergey..
FYI, I have been closely looking at dp_counter_mm.q failure,
which only happens during pre-commit tests (not on Mac laptop or our cn105
CentOS cluster).
Finally I found out the root cause of the problem which is related to "union
all" so will add the fix of that failure
shortly but NOT a big change.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268121#comment-16268121 ] Deepak Jaiswal edited comment on HIVE-18157 at 11/28/17 5:01 AM: - I dont think so, This is the JIRA which caused this, https://issues.apache.org/jira/browse/HIVE-16557 I verified that it was never committed to branch-2 was (Author: djaiswal): I dont think so, This is the JIRA which caused this, https://issues.apache.org/jira/browse/HIVE-16557 > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268121#comment-16268121 ] Deepak Jaiswal commented on HIVE-18157: --- I dont think so, This is the JIRA which caused this, https://issues.apache.org/jira/browse/HIVE-16557 > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type
[
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268116#comment-16268116
]
Hive QA commented on HIVE-17972:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899520/HIVE-17972.002.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11416 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=282)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8012/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8012/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8012/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899520 - PreCommit-HIVE-Build
> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch, HIVE-17972.002.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to
> improve the performance when the query with map type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268111#comment-16268111
]
Eugene Koifman commented on HIVE-17856:
---
[~sershe], could you hold off checking this in - I'd like to look at it
tomorrow.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268102#comment-16268102 ] Vihang Karajgaonkar commented on HIVE-18157: Hi [~djaiswal] Do you know if this issue happens on branch-2 as well? Thanks! > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17970) MM LOAD DATA with OVERWRITE doesn't use base_n directory concept
[
https://issues.apache.org/jira/browse/HIVE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268072#comment-16268072
]
Sergey Shelukhin commented on HIVE-17970:
-
cc [~hagleitn] [~steveyeom2017] another MM gap (which I guess was expected),
followup from HIVE-17856
> MM LOAD DATA with OVERWRITE doesn't use base_n directory concept
>
>
> Key: HIVE-17970
> URL: https://issues.apache.org/jira/browse/HIVE-17970
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
> Labels: mm-gap-2
>
> Judging by
> {code:java}
> Hive.loadTable(Path loadPath, String tableName, LoadFileType loadFileType,
> boolean isSrcLocal,
> boolean isSkewedStoreAsSubdir, boolean isAcid, boolean
> hasFollowingStatsTask,
> Long txnId, int stmtId, boolean isMmTable)
> {code}
> LOAD DATA with OVERWRITE will delete all existing data then write new data
> into the table. This logic makes sense for non-acid tables but for Acid/MM
> it should work like INSERT OVERWRITE statement and write new data to base_n/.
> This way the lock manager can be used to either get an X lock for IOW and
> thus block all readers or let it run with SemiShared and let readers continue
> and make the system more concurrent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17970) MM LOAD DATA with OVERWRITE doesn't use base_n directory concept
[
https://issues.apache.org/jira/browse/HIVE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17970:
Labels: mm-gap-2 (was: )
> MM LOAD DATA with OVERWRITE doesn't use base_n directory concept
>
>
> Key: HIVE-17970
> URL: https://issues.apache.org/jira/browse/HIVE-17970
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
> Labels: mm-gap-2
>
> Judging by
> {code:java}
> Hive.loadTable(Path loadPath, String tableName, LoadFileType loadFileType,
> boolean isSrcLocal,
> boolean isSkewedStoreAsSubdir, boolean isAcid, boolean
> hasFollowingStatsTask,
> Long txnId, int stmtId, boolean isMmTable)
> {code}
> LOAD DATA with OVERWRITE will delete all existing data then write new data
> into the table. This logic makes sense for non-acid tables but for Acid/MM
> it should work like INSERT OVERWRITE statement and write new data to base_n/.
> This way the lock manager can be used to either get an X lock for IOW and
> thus block all readers or let it run with SemiShared and let readers continue
> and make the system more concurrent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type
[
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268071#comment-16268071
]
Hive QA commented on HIVE-17972:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8012/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8012/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch, HIVE-17972.002.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to
> improve the performance when the query with map type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17856:
Attachment: HIVE-17856.15.patch
The same patch for QA
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-17856:
---
Assignee: Steve Yeom (was: Sergey Shelukhin)
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-17856:
---
Assignee: Sergey Shelukhin (was: Steve Yeom)
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch,
> HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch,
> HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268069#comment-16268069
]
Sergey Shelukhin commented on HIVE-17856:
-
+1 one small nit, I can fix on commit if it's ok.
Is dp_counter_mm failure related? The results have expired so I cannot tell
what it was. Let me reattach for another run to dbl check...
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch,
> HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch,
> HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Attachment: HIVE-18134.02.patch Added ordering to the test > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.01.patch, HIVE-18134.02.patch, > HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268052#comment-16268052
]
Hive QA commented on HIVE-18147:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899513/HIVE-18147.4.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal]
(batchId=9)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8011/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8011/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8011/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899513 - PreCommit-HIVE-Build
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch,
> HIVE-18147.3.patch, HIVE-18147.4.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is started, the port can get used by some other service
> resulting in test failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268048#comment-16268048 ] Sergey Shelukhin commented on HIVE-18076: - Cannot repro the timeout. No error in the log either... too bad the thing doesn't dump entire thread state. > killquery doesn't actually work for non-trigger WM kills > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.03.patch, HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18076: Attachment: HIVE-18076.03.patch > killquery doesn't actually work for non-trigger WM kills > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.03.patch, HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268014#comment-16268014
]
Hive QA commented on HIVE-18147:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch svr passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} itests/util: The patch generated 0 new + 21
unchanged - 1 fixed = 21 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8011/yetus/patch-asflicense-problems.txt
|
| modules | C: metastore service hcatalog/core hcatalog/webhcat/svr
itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8011/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch,
> HIVE-18147.3.patch, HIVE-18147.4.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267991#comment-16267991
]
Hive QA commented on HIVE-18134:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899509/HIVE-18134.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8010/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8010/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8010/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899509 - PreCommit-HIVE-Build
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.01.patch, HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[ https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-17528: Attachment: HIVE-17528.8.patch > Add more q-tests for Hive-on-Spark with Parquet vectorized reader > - > > Key: HIVE-17528 > URL: https://issues.apache.org/jira/browse/HIVE-17528 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Ferdinand Xu > Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch, > HIVE-17528.3.patch, HIVE-17528.4.patch, HIVE-17528.5.patch, > HIVE-17528.7.patch, HIVE-17528.8.patch, HIVE-17528.patch > > > Most of the vectorization related q-tests operate on ORC tables using Tez. It > would be good to add more coverage on a different combination of engine and > file-format. We can model existing q-tests using parquet tables and run it > using TestSparkCliDriver -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Attachment: HIVE-17361.24.patch > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.23.patch, HIVE-17361.24.patch, HIVE-17361.3.patch, > HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Schema is not too different). > * It does check that _InputFormat_S are compatible. > * Bucketed (and thus sorted) tables don't support Load Data (but only if > hive.strict.checks.bucketing=true (default)). Will keep this restriction for > Acid. > * Load Data supports OVERWRITE clause > * What happens to file permissions/ownership: rename vs copy differences > \\ > The implementation will follow the same idea as in HIVE-14988 and use a > base_N/ dir for OVERWRITE clause. > \\ > How is minor compaction going to handle delta/base with original files? > Since delta_8_8/_meta_data is created before files are moved, delta_8_8 > becomes visible before it's populated. Is that an issue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18159) Vectorization: Support Map type in MapWork
[ https://issues.apache.org/jira/browse/HIVE-18159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma reassigned HIVE-18159: --- > Vectorization: Support Map type in MapWork > -- > > Key: HIVE-18159 > URL: https://issues.apache.org/jira/browse/HIVE-18159 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma > > Support Complex Types in vectorization is finished in HIVE-16589, but Map > type is still not support in MapWork. This ticket is target to support it for > MapWork when vectorization is enable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18157: -- Attachment: HIVE-18157.1.patch Initial patch. Thanks [~gopalv] for providing a test case. [~gopalv] [~mmccline] can you please review? > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18157.1.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18158) Remove OrcRawRecordMerger.ReaderPairAcid.statementId
[ https://issues.apache.org/jira/browse/HIVE-18158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18158: - > Remove OrcRawRecordMerger.ReaderPairAcid.statementId > > > Key: HIVE-18158 > URL: https://issues.apache.org/jira/browse/HIVE-18158 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > * Need to get rid of this since we can always get this from the row > itself in Acid 2.0. > * For Acid 1.0, statementId == 0 in all deltas because both > multi-statement txns and > * Split Upate are only available in test mode so there is nothing can > create a > * deltas_x_x_M with M > 0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18157: -- Status: Patch Available (was: In Progress) > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18043) Vectorization: Support List type in MapWork
[ https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18043: Attachment: HIVE-18043.005.patch [~Ferd], the patch is updated and only one warning of Yetus is left. It's about variable name, current name is "listTypePattern" which is the same format as "charTypePattern", "varcharTypePattern", "structTypePattern", etc. So, I suggest to keep all these with the same format, and fix it if necessary in a separated ticket. > Vectorization: Support List type in MapWork > --- > > Key: HIVE-18043 > URL: https://issues.apache.org/jira/browse/HIVE-18043 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch, > HIVE-18043.003.patch, HIVE-18043.004.patch, HIVE-18043.005.patch > > > Support Complex Types in vectorization is finished in HIVE-16589, but List > type is still not support in MapWork. It should be supported to improve the > performance when vectorization is enable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18157 started by Deepak Jaiswal. - > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267955#comment-16267955
]
Hive QA commented on HIVE-18134:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore: The patch generated 1 new + 477
unchanged - 0 fixed = 478 total (was 477) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8010/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8010/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8010/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.01.patch, HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-18157: - > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18156) Provide smooth migration path for CTAS when time column is not with timezone
[
https://issues.apache.org/jira/browse/HIVE-18156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-18156:
-
> Provide smooth migration path for CTAS when time column is not with timezone
> -
>
> Key: HIVE-18156
> URL: https://issues.apache.org/jira/browse/HIVE-18156
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>
> Currently, default recommend CTAS and most legacy documentation does not
> specify that __time column needs to be with timezone. Thus the CTAS will fail
> with
> {code}
> 2017-11-27T17:13:10,241 ERROR [e5f708c8-df4e-41a4-b8a1-d18ac13123d2 main]
> ql.Driver: FAILED: SemanticException No column with timestamp with local
> time-zone type on query result; one column should be of timestamp with local
> time-zone type
> org.apache.hadoop.hive.ql.parse.SemanticException: No column with timestamp
> with local time-zone type on query result; one column should be of timestamp
> with local time-zone type
> at
> org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionTimeGranularityOptimizer$SortedDynamicPartitionProc.getGranularitySelOp(SortedDynPartitionTimeGranularityOptimizer.java:242)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionTimeGranularityOptimizer$SortedDynamicPartitionProc.process(SortedDynPartitionTimeGranularityOptimizer.java:163)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionTimeGranularityOptimizer.transform(SortedDynPartitionTimeGranularityOptimizer.java:103)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11683)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:298)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:592)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1589)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1356)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1346)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:342)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1300)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1274)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type
[ https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267942#comment-16267942 ] Ferdinand Xu commented on HIVE-17972: - Thanks [~colinma] for the patch. +1 pending on the test > Implement Parquet vectorization reader for Map type > --- > > Key: HIVE-17972 > URL: https://issues.apache.org/jira/browse/HIVE-17972 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-17972.001.patch, HIVE-17972.002.patch > > > Parquet vectorized reader can't support map type, it should be supported to > improve the performance when the query with map type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17972) Implement Parquet vectorization reader for Map type
[ https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-17972: Attachment: HIVE-17972.002.patch [~Ferd], thanks for the review, the patch is updated according to your comments. > Implement Parquet vectorization reader for Map type > --- > > Key: HIVE-17972 > URL: https://issues.apache.org/jira/browse/HIVE-17972 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-17972.001.patch, HIVE-17972.002.patch > > > Parquet vectorized reader can't support map type, it should be supported to > improve the performance when the query with map type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18154) Acid Load Data/Insert with Overwrite in multi statement transactions
[
https://issues.apache.org/jira/browse/HIVE-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18154:
--
Description:
Consider:
{noformat}
START TRANSACTION
insert into T values(1,2),(3,4)
load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into table T
update T set a = 0 where a = 6
COMMIT
{noformat}
So what we should have on disk is
{noformat}
├── base_028
│  ├── 00_0
│  └── _metadata_acid
├── delete_delta_028_028_0002
│  └── bucket_0
├── delta_028_028_
│  └── bucket_0
└── delta_028_028_0002
└── bucket_0
{noformat}
where base_28 is from overwrite, delta_028_028_ from 1st insert nad
delta_028_028_0002/delete_delta_028_028_0002 is from update.
AcidUtils.getAcidState() only returns base_28 thinking that all other deltas
are included in it - not what we want here.
Same applies for Insert Overwrite.
The simple way to get correct behavior is to disallow commands with Overwrite
clause in multi-statement txns.
was:
Consider:
{noformat}
START TRANSACTION
insert into T values(1,2),(3,4)
load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into table T
update T set a = 0 where a = 6
COMMIT
{noformat}
So what we should have on disk is
{noformat}
├── base_028
│  ├── 00_0
│  └── _metadata_acid
├── delete_delta_028_028_0002
│  └── bucket_0
├── delta_028_028_
│  └── bucket_0
└── delta_028_028_0002
└── bucket_0
{noformat}
where base_28 is from overwrite, delta_028_028_ from 1st insert nad
delta_028_028_0002/delete_delta_028_028_0002 is from update.
AcidUtils.getAcidState() only returns base_28 thinking that all other deltas
are included in it - not what we want here.
The simple way to get correct behavior is to disallow commands with Overwrite
clause in multi-statement txns.
> Acid Load Data/Insert with Overwrite in multi statement transactions
>
>
> Key: HIVE-18154
> URL: https://issues.apache.org/jira/browse/HIVE-18154
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider:
> {noformat}
> START TRANSACTION
> insert into T values(1,2),(3,4)
> load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into
> table T
> update T set a = 0 where a = 6
> COMMIT
> {noformat}
> So what we should have on disk is
> {noformat}
> ├── base_028
> │  ├── 00_0
> │  └── _metadata_acid
> ├── delete_delta_028_028_0002
> │  └── bucket_0
> ├── delta_028_028_
> │  └── bucket_0
> └── delta_028_028_0002
> └── bucket_0
> {noformat}
> where base_28 is from overwrite, delta_028_028_ from 1st insert
> nad delta_028_028_0002/delete_delta_028_028_0002 is from
> update.
> AcidUtils.getAcidState() only returns base_28 thinking that all other deltas
> are included in it - not what we want here.
> Same applies for Insert Overwrite.
> The simple way to get correct behavior is to disallow commands with Overwrite
> clause in multi-statement txns.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267932#comment-16267932
]
Hive QA commented on HIVE-18076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899506/HIVE-18076.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing
(batchId=282)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8009/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8009/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8009/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899506 - PreCommit-HIVE-Build
> killquery doesn't actually work for non-trigger WM kills
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch,
> HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18155) HiveTxnManager.getWriteIdAndIncrement() referred to as statementId
[ https://issues.apache.org/jira/browse/HIVE-18155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18155: - > HiveTxnManager.getWriteIdAndIncrement() referred to as statementId > -- > > Key: HIVE-18155 > URL: https://issues.apache.org/jira/browse/HIVE-18155 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > HiveTxnManager.getWriteIdAndIncrement() referred to as statementId in > AcidUtils and many other places. It should be renamed - it currently counts > _FileSinkOperator_ instances in a transaction (not just statements) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18154) Acid Load Data/Insert with Overwrite in multi statement transactions
[
https://issues.apache.org/jira/browse/HIVE-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-18154:
-
> Acid Load Data/Insert with Overwrite in multi statement transactions
>
>
> Key: HIVE-18154
> URL: https://issues.apache.org/jira/browse/HIVE-18154
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider:
> {noformat}
> START TRANSACTION
> insert into T values(1,2),(3,4)
> load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into
> table T
> update T set a = 0 where a = 6
> COMMIT
> {noformat}
> So what we should have on disk is
> {noformat}
> ├── base_028
> │  ├── 00_0
> │  └── _metadata_acid
> ├── delete_delta_028_028_0002
> │  └── bucket_0
> ├── delta_028_028_
> │  └── bucket_0
> └── delta_028_028_0002
> └── bucket_0
> {noformat}
> where base_28 is from overwrite, delta_028_028_ from 1st insert
> nad delta_028_028_0002/delete_delta_028_028_0002 is from
> update.
> AcidUtils.getAcidState() only returns base_28 thinking that all other deltas
> are included in it - not what we want here.
> The simple way to get correct behavior is to disallow commands with Overwrite
> clause in multi-statement txns.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18043) Vectorization: Support List type in MapWork
[ https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267915#comment-16267915 ] Ferdinand Xu commented on HIVE-18043: - LGTM, please help fix the issue pointed by Yetus. > Vectorization: Support List type in MapWork > --- > > Key: HIVE-18043 > URL: https://issues.apache.org/jira/browse/HIVE-18043 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch, > HIVE-18043.003.patch, HIVE-18043.004.patch > > > Support Complex Types in vectorization is finished in HIVE-16589, but List > type is still not support in MapWork. It should be supported to improve the > performance when vectorization is enable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267878#comment-16267878
]
Hive QA commented on HIVE-18076:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 28 new + 192 unchanged - 24
fixed = 220 total (was 216) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8009/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8009/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-tez ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8009/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> killquery doesn't actually work for non-trigger WM kills
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch,
> HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18147: --- Attachment: HIVE-18147.4.patch > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch, HIVE-18147.4.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267848#comment-16267848
]
Hive QA commented on HIVE-16708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899492/HIVE-16708.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11416 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8008/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8008/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8008/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899492 - PreCommit-HIVE-Build
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch,
> HIVE-16708.03.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
> https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105
> Let me know If I am missing something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Attachment: HIVE-18134.01.patch Rebased on top of the commands patch that already had most of the fixes. Only added the disable check and tests > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.01.patch, HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267798#comment-16267798
]
Hive QA commented on HIVE-16708:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a21742e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8008/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore itests/hive-minikdc U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8008/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch,
> HIVE-16708.03.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
>
[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18076: Attachment: HIVE-18076.02.patch Fixing the reopen path. Not sure why reopen triggers in the first place, but it looks like this has been triggering sporadically in various runs. I'll file a follow up to either refactor how TezTask handles reopen, so it doesn't rely on it being the same object, or change WM to preserve the same object if the former is too difficult. > killquery doesn't actually work for non-trigger WM kills, or the error > message is not propagated > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18076: Summary: killquery doesn't actually work for non-trigger WM kills (was: killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated) > killquery doesn't actually work for non-trigger WM kills > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.02.patch, > HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18029) beeline - support proper usernames based on the URL arg
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the review!
> beeline - support proper usernames based on the URL arg
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18029.patch
>
>
> Update: looks like the argument connection URL is not handled consistently
> with the connect command one, and the latter passes on user name that's
> entered in the prompt correctly.
> So,
> {noformat}
> !connect (url) => prompt; the username on HS2 side is whatever is entered in
> the prompt
> beeline -u (url) => anonymous (no prompt)
> !connect (url);user=foo => foo
> beeline -u (url);user=foo => anonymous
> beeline -n foo -u (url with or without the user) => foo
> {noformat}
> I'm going to add support for extracting the user from the -u argument,
> similar to connect argument
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267770#comment-16267770
]
Hive QA commented on HIVE-18147:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899491/HIVE-18147.3.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.metastore.TestMetaStoreListenersError.testEventListenerException
(batchId=218)
org.apache.hadoop.hive.metastore.TestMetaStoreListenersError.testInitListenerException
(batchId=218)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8007/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8007/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8007/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899491 - PreCommit-HIVE-Build
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch,
> HIVE-18147.3.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is started, the port can get used by some other service
> resulting in test failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18029) beeline - support proper usernames based on the URL arg
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Summary: beeline - support proper usernames based on the URL arg (was:
beeline - support proper usernames based on URL)
> beeline - support proper usernames based on the URL arg
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Update: looks like the argument connection URL is not handled consistently
> with the connect command one, and the latter passes on user name that's
> entered in the prompt correctly.
> So,
> {noformat}
> !connect (url) => prompt; the username on HS2 side is whatever is entered in
> the prompt
> beeline -u (url) => anonymous (no prompt)
> !connect (url);user=foo => foo
> beeline -u (url);user=foo => anonymous
> beeline -n foo -u (url with or without the user) => foo
> {noformat}
> I'm going to add support for extracting the user from the -u argument,
> similar to connect argument
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Description:
Update: looks like the argument connection URL is not handled consistently
with the connect command one, and the latter passes on user name that's entered
in the prompt correctly.
So,
{noformat}
!connect (url) => prompt; the username on HS2 side is whatever is entered in
the prompt
beeline -u (url) => anonymous (no prompt)
!connect (url);user=foo => foo
beeline -u (url);user=foo => anonymous
beeline -n foo -u (url with or without the user) => foo
{noformat}
I'm going to add support for extracting the user from the -u argument, similar
to connect argument
was:
Right now what happens on unsecure cluster with doAs=false (not sure which one
is to blame - didn't look into it, maybe both) is {noformat}
2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
tez.WorkloadManagerFederation: Getting a WM session for anonymous
{noformat}
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Update: looks like the argument connection URL is not handled consistently
> with the connect command one, and the latter passes on user name that's
> entered in the prompt correctly.
> So,
> {noformat}
> !connect (url) => prompt; the username on HS2 side is whatever is entered in
> the prompt
> beeline -u (url) => anonymous (no prompt)
> !connect (url);user=foo => foo
> beeline -u (url);user=foo => anonymous
> beeline -n foo -u (url with or without the user) => foo
> {noformat}
> I'm going to add support for extracting the user from the -u argument,
> similar to connect argument
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267743#comment-16267743
]
Vaibhav Gumashta commented on HIVE-18029:
-
+1
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18073) AM may assert when its guaranteed task count is reduced
[ https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18073: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! > AM may assert when its guaranteed task count is reduced > --- > > Key: HIVE-18073 > URL: https://issues.apache.org/jira/browse/HIVE-18073 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18073.patch > > > Sometimes it asserts that it doesn't have so many ducks to give away. This > should never happen, need to debug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18073) AM may assert when its guaranteed task count is reduced
[ https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267722#comment-16267722 ] Sergey Shelukhin edited comment on HIVE-18073 at 11/27/17 10:39 PM: Committed to master. Thanks for the review! was (Author: sershe): Committed to master. Thanks for the reviews! > AM may assert when its guaranteed task count is reduced > --- > > Key: HIVE-18073 > URL: https://issues.apache.org/jira/browse/HIVE-18073 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18073.patch > > > Sometimes it asserts that it doesn't have so many ducks to give away. This > should never happen, need to debug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267713#comment-16267713
]
Hive QA commented on HIVE-18147:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch svr passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} itests/util: The patch generated 0 new + 21
unchanged - 1 fixed = 21 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be1f847 |
| Default Java | 1.8.0_111 |
| modules | C: metastore service hcatalog/core hcatalog/webhcat/svr
itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8007/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch,
> HIVE-18147.3.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is started, the port can get used by some other service
> resulting in test failure.
--
This message was sent by
[jira] [Updated] (HIVE-18073) AM may assert when its guaranteed task count is reduced
[ https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18073: Summary: AM may assert when its guaranteed task count is reduced (was: AM may assert when duck count for it is reduced) > AM may assert when its guaranteed task count is reduced > --- > > Key: HIVE-18073 > URL: https://issues.apache.org/jira/browse/HIVE-18073 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18073.patch > > > Sometimes it asserts that it doesn't have so many ducks to give away. This > should never happen, need to debug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17954: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. The changes to output are ordering-related. Perhaps it makes sense to make the views sorted alphabetically by default. Thanks for all the work on the patch! > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Fix For: 3.0.0 > > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[
https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267677#comment-16267677
]
Hive QA commented on HIVE-17954:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899484/HIVE-17954.10.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorPartitionWithBase
(batchId=264)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8006/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8006/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8006/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899484 - PreCommit-HIVE-Build
> Implement pool, user, group and trigger to pool management API's.
> -
>
> Key: HIVE-17954
> URL: https://issues.apache.org/jira/browse/HIVE-17954
> Project: Hive
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch,
> HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch,
> HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch,
> HIVE-17954.09.patch, HIVE-17954.10.patch
>
>
> Implement the following commands:
> -- Pool management.
> CREATE POOL `resource_plan`.`pool_path` WITH
> ALLOC_FRACTION=`fraction`,
> QUERY_PARALLELISM=`parallelism`,
> SCHEDULING_POLICY=`policy`;
> ALTER POOL `resource_plan`.`pool_path` SET
> PATH = `new_path`,
> ALLOC_FRACTION = `fraction`,
> QUERY_PARALLELISM = `parallelism`,
> SCHEDULING_POLICY = `policy`;
> DROP POOL `resource_plan`.`pool_path`;
> -- Adding triggers to pools.
> ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`;
> ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`;
> -- User/Group to pool mappings.
> CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name`
> TO `pool_path` WITH ORDERING `order_no`;
> DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18151) LLAP external client: Better error message propagation during submission failures
[ https://issues.apache.org/jira/browse/HIVE-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267675#comment-16267675 ] Sergey Shelukhin commented on HIVE-18151: - +1 > LLAP external client: Better error message propagation during submission > failures > - > > Key: HIVE-18151 > URL: https://issues.apache.org/jira/browse/HIVE-18151 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-18151.1.patch > > > During failed submissions, the original error message should be added to the > error message that is eventually propagated to the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15344) Spelling errors in logging and exceptions for metastore and service directories
[ https://issues.apache.org/jira/browse/HIVE-15344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267619#comment-16267619 ] Grant Sohn commented on HIVE-15344: --- Thanks for the +1, Ashutosh! > Spelling errors in logging and exceptions for metastore and service > directories > --- > > Key: HIVE-15344 > URL: https://issues.apache.org/jira/browse/HIVE-15344 > Project: Hive > Issue Type: Bug > Components: CLI, Metastore, Server Infrastructure >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-15344.1.patch > > > More spelling errors in logging and exception messages. > verififcation -> verification > indexxes -> indexes > resouce -> resource -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15343) Spelling errors in logging and exceptions for beeline, common, hbase-handler, hcatalog, llap-server, orc, serde and shims
[ https://issues.apache.org/jira/browse/HIVE-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267621#comment-16267621 ] Grant Sohn commented on HIVE-15343: --- Thanks for the +1, Ashutosh! > Spelling errors in logging and exceptions for beeline, common, hbase-handler, > hcatalog, llap-server, orc, serde and shims > - > > Key: HIVE-15343 > URL: https://issues.apache.org/jira/browse/HIVE-15343 > Project: Hive > Issue Type: Bug > Components: Beeline, HBase Handler, HCatalog, llap, ORC, > Serializers/Deserializers, Shims >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-15343.1.patch > > > There are a set of misspelled words in the logs and exceptions. > Wtaited -> Waited > prioroty -> priority -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18073) AM may assert when duck count for it is reduced
[ https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267609#comment-16267609 ] Prasanth Jayachandran commented on HIVE-18073: -- +1 > AM may assert when duck count for it is reduced > --- > > Key: HIVE-18073 > URL: https://issues.apache.org/jira/browse/HIVE-18073 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18073.patch > > > Sometimes it asserts that it doesn't have so many ducks to give away. This > should never happen, need to debug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[
https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267599#comment-16267599
]
Hive QA commented on HIVE-17954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore: The patch generated 28 new +
1312 unchanged - 0 fixed = 1340 total (was 1312) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} metastore: The patch generated 35 new + 1011 unchanged
- 0 fixed = 1046 total (was 1011) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 66 new + 1277 unchanged - 14
fixed = 1343 total (was 1291) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 136 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be1f847 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus/diff-checkstyle-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore metastore ql itests/hcatalog-unit
itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8006/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement pool, user, group and trigger to pool management API's.
> -
>
> Key: HIVE-17954
> URL: https://issues.apache.org/jira/browse/HIVE-17954
> Project: Hive
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch,
> HIVE-17954.03.patch,
[jira] [Updated] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16708: --- Attachment: HIVE-16708.03.patch Attaching the patch again to re-rerun the HiveQA > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch, > HIVE-16708.03.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267583#comment-16267583 ] Vihang Karajgaonkar commented on HIVE-18147: +1 LGTM. Hi [~janulatha] Can you please fix the checkstyle issues so that Yetus is happy ;) ./itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:39:import org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge;:8: warning: Unused import - org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge. ./itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:348: Exception HS2Exception = null;:15: warning: Name 'HS2Exception' must match pattern '^[a-z][a-zA-Z0-9]*$'. ./itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java:349: boolean HS2Started = false;:13: warning: Name 'HS2Started' must match pattern '^[a-z][a-zA-Z0-9]*$'. > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18147: --- Attachment: HIVE-18147.3.patch Fixed "checkstyle" issue reported by Hive QA. > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch, > HIVE-18147.3.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267509#comment-16267509 ] Sergey Shelukhin commented on HIVE-18134: - Will commit after HIVE-17954 and a rebase... still need an official +1 too :) Any path should be acceptable. I may change it as part of this patch to not require quotes and support identifier(.identifier)* or smth like that > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267527#comment-16267527
]
Hive QA commented on HIVE-18147:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899476/HIVE-18147.2.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11412 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.metastore.TestMetaStoreListenersError.testEventListenerException
(batchId=218)
org.apache.hadoop.hive.metastore.TestMetaStoreListenersError.testInitListenerException
(batchId=218)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8005/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8005/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8005/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899476 - PreCommit-HIVE-Build
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is started, the port can get used by some other service
> resulting in test failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18037) Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x
[ https://issues.apache.org/jira/browse/HIVE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267523#comment-16267523 ] Gour Saha commented on HIVE-18037: -- bq. Can you refer to YARN packages by a separate version for now? We cannot bring in only the new packages with 3.1 version. There are a lot of dependent changes in core Hadoop packages as well and hence the new packages will not work. > Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x > > > Key: HIVE-18037 > URL: https://issues.apache.org/jira/browse/HIVE-18037 > Project: Hive > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha > Fix For: 3.0.0 > > Attachments: HIVE-18037.001.patch > > > Apache Slider has been migrated to Hadoop-3.x and is referred to as YARN > Service (YARN-4692). Most of the classic Slider features are now going to be > supported in a first-class manner by core YARN. It includes several new > features like a RESTful API. Command line equivalents of classic Slider are > supported by YARN Service as well. > This jira will take care of all changes required to Slider LLAP packaging and > scripts to make it work against Hadoop 3.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18151) LLAP external client: Better error message propagation during submission failures
[ https://issues.apache.org/jira/browse/HIVE-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267510#comment-16267510 ] Jason Dere commented on HIVE-18151: --- [~sershe] quick review? Just a one-liner to include the exception message in an error string. > LLAP external client: Better error message propagation during submission > failures > - > > Key: HIVE-18151 > URL: https://issues.apache.org/jira/browse/HIVE-18151 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-18151.1.patch > > > During failed submissions, the original error message should be added to the > error message that is eventually propagated to the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18151) LLAP external client: Better error message propagation during submission failures
[ https://issues.apache.org/jira/browse/HIVE-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18151: -- Attachment: HIVE-18151.1.patch > LLAP external client: Better error message propagation during submission > failures > - > > Key: HIVE-18151 > URL: https://issues.apache.org/jira/browse/HIVE-18151 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-18151.1.patch > > > During failed submissions, the original error message should be added to the > error message that is eventually propagated to the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18151) LLAP external client: Better error message propagation during submission failures
[ https://issues.apache.org/jira/browse/HIVE-18151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-18151: - > LLAP external client: Better error message propagation during submission > failures > - > > Key: HIVE-18151 > URL: https://issues.apache.org/jira/browse/HIVE-18151 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > > During failed submissions, the original error message should be added to the > error message that is eventually propagated to the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267450#comment-16267450 ] Sergey Shelukhin commented on HIVE-17954: - +1 pending a test run > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17954: --- Assignee: Sergey Shelukhin (was: Harish Jaiprakash) > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Sergey Shelukhin > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17954: --- Assignee: Harish Jaiprakash (was: Sergey Shelukhin) > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17954: Attachment: HIVE-17954.10.patch The same patch to retry HiveQA > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Sergey Shelukhin > Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, > HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, > HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch, > HIVE-17954.09.patch, HIVE-17954.10.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION=`fraction`, > QUERY_PARALLELISM=`parallelism`, > SCHEDULING_POLICY=`policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Adding triggers to pools. > ALTER POOL `resource_plan`.`pool_path` ADD TRIGGER `trigger_name`; > ALTER POOL `resource_plan`.`pool_path` DROP TRIGGER `trigger_name`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18091) Failing tests of itests/qtest-spark and itests/hive-unit on branch-1
[
https://issues.apache.org/jira/browse/HIVE-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267441#comment-16267441
]
Sergey Shelukhin commented on HIVE-18091:
-
+1
> Failing tests of itests/qtest-spark and itests/hive-unit on branch-1
>
>
> Key: HIVE-18091
> URL: https://issues.apache.org/jira/browse/HIVE-18091
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: Daniel Voros
>Assignee: Daniel Voros
> Attachments: HIVE-18091.1-branch-1.patch, HIVE-18091.2-branch-1.patch
>
>
> Seen this while looking at ptest results for HIVE-17947 but is probably an
> older issue.
> Tests under itests/qtest-spark and itests/hive-unit fail when trying to
> execute the download-spark plugin with:
> {code}
> [INFO]
> [INFO]
>
> [INFO] Building Hive Integration - Unit Tests 1.3.0-SNAPSHOT
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
> hive-it-unit ---
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-unit ---
> [INFO] Executing tasks
> main:
> [exec] + /bin/pwd
> [exec] + BASE_DIR=./target
> [exec] + HIVE_ROOT=./target/../../../
> [exec] + DOWNLOAD_DIR=./../thirdparty
> [exec] + mkdir -p ./../thirdparty
> [exec]
> /home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit
> [exec] + download
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> spark
> [exec] +
> url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + finalName=spark
> [exec] ++ basename
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + tarName=spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + rm -rf ./target/spark
> [exec] + [[ ! -f
> ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz ]]
> [exec] + tar -zxf
> ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz -C ./target
> [exec] + mv ./target/spark-1.5.0-bin-hadoop2-without-hive ./target/spark
> [exec] + cp -f ./target/../../..//data/conf/spark/log4j.properties
> ./target/spark/conf/
> [exec] + sed '/package /d'
> /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
> [exec] sed: can't read
> /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java:
> No such file or directory
> [exec] + javac -cp
> /data/hiveptest/working/maven/org/apache/hive/hive-exec/1.3.0-SNAPSHOT/hive-exec-1.3.0-SNAPSHOT.jar
> /tmp/UDFExampleAdd.java -d /tmp
> [exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
> [exec] /tmp/UDFExampleAdd.class : no such file or directory
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 8.376s
> [INFO] Finished at: Mon Nov 06 22:29:39 UTC 2017
> [INFO] Final Memory: 18M/241M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (download-spark) on
> project hive-it-unit: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part ... dir="/home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit"
> executable="bash" failonerror="true">... @ 4:141 in
> /home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit/target/antrun/build-main.xml
> {code}
> {{mvn antrun:run@download-spark}} passes when run locally, so I guess it
> might be an issue with the way wer're executing ptests.
> Full list of classes with the same error:
> {code}
> TestAcidOnTez
> TestAdminUser
> TestAuthorizationPreEventListener
> TestAuthzApiEmbedAuthorizerInEmbed
> TestAuthzApiEmbedAuthorizerInRemote
> TestBeeLineWithArgs
> TestCLIAuthzSessionContext
> TestClearDanglingScratchDir
> TestClientSideAuthorizationProvider
> TestCompactor
> TestCreateUdfEntities
> TestCustomAuthentication
> TestDBTokenStore
> TestDDLWithRemoteMetastoreSecondNamenode
> TestDynamicSerDe
> TestEmbeddedHiveMetaStore
> TestEmbeddedThriftBinaryCLIService
> TestFilterHooks
> TestFolderPermissions
> TestHS2AuthzContext
> TestHS2AuthzSessionContext
> TestHS2ClearDanglingScratchDir
>
[jira] [Commented] (HIVE-18037) Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x
[ https://issues.apache.org/jira/browse/HIVE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267426#comment-16267426 ] Sergey Shelukhin commented on HIVE-18037: - cc [~ashutoshc] should we upgrade to Hadoop 3.1? I actually don't think we can upgrade right now, iirc there were some other breaking changes there. Can you refer to YARN packages by a separate version for now? > Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x > > > Key: HIVE-18037 > URL: https://issues.apache.org/jira/browse/HIVE-18037 > Project: Hive > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha > Fix For: 3.0.0 > > Attachments: HIVE-18037.001.patch > > > Apache Slider has been migrated to Hadoop-3.x and is referred to as YARN > Service (YARN-4692). Most of the classic Slider features are now going to be > supported in a first-class manner by core YARN. It includes several new > features like a RESTful API. Command line equivalents of classic Slider are > supported by YARN Service as well. > This jira will take care of all changes required to Slider LLAP packaging and > scripts to make it work against Hadoop 3.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[
https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267410#comment-16267410
]
Hive QA commented on HIVE-18147:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/util: The patch generated 3 new + 21 unchanged
- 1 fixed = 24 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be1f847 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8005/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: metastore service hcatalog/core hcatalog/webhcat/svr
itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8005/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tests can fail with java.net.BindException: Address already in use
> --
>
> Key: HIVE-18147
> URL: https://issues.apache.org/jira/browse/HIVE-18147
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch
>
>
> Between the time a new port is picked up to start HMS or HS2, and the time
> when HMS or HS2 is started, the port can get used by some other service
> resulting in test failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267407#comment-16267407
]
Vihang Karajgaonkar commented on HIVE-16708:
TestTriggersWorkloadManager is failing due to {{java.lang.Exception: test timed
out after 6 milliseconds}} which seems like some setup issue. Its running
fine for me locally with the patch. The other tests have been failing for a
while now.
{noformat}
[INFO] Running org.apache.hive.jdbc.TestTriggersWorkloadManager
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 170.925
s - in org.apache.hive.jdbc.TestTriggersWorkloadManager
[INFO]
[INFO] Results:
{noformat}
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
> https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105
> Let me know If I am missing something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267365#comment-16267365
]
Hive QA commented on HIVE-16708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899462/HIVE-16708.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11416 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8004/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8004/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12899462 - PreCommit-HIVE-Build
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
> https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105
> Let me know If I am missing something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18111) Fix temp path for Spark DPP sink
[
https://issues.apache.org/jira/browse/HIVE-18111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267342#comment-16267342
]
Sahil Takiar commented on HIVE-18111:
-
LGTM, anyway to add a test for this?
> Fix temp path for Spark DPP sink
>
>
> Key: HIVE-18111
> URL: https://issues.apache.org/jira/browse/HIVE-18111
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-18111.1.patch, HIVE-18111.2.patch,
> HIVE-18111.3.patch
>
>
> Before HIVE-17877, each DPP sink has only one target work. The output path of
> a DPP work is {{TMP_PATH/targetWorkId/dppWorkId}}. When we do the pruning,
> each map work reads DPP outputs under {{TMP_PATH/targetWorkId}}.
> After HIVE-17877, each DPP sink can have multiple target works. It's possible
> that a map work needs to read DPP outputs from multiple
> {{TMP_PATH/targetWorkId}}. To solve this, I think we can have a DPP output
> path specific to each query, e.g. {{QUERY_TMP_PATH/dpp_output}}. Each DPP
> work outputs to {{QUERY_TMP_PATH/dpp_output/dppWorkId}}. And each map work
> reads from {{QUERY_TMP_PATH/dpp_output}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
[ https://issues.apache.org/jira/browse/HIVE-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18147: --- Attachment: HIVE-18147.2.patch Addressed comments from [~asherman] > Tests can fail with java.net.BindException: Address already in use > -- > > Key: HIVE-18147 > URL: https://issues.apache.org/jira/browse/HIVE-18147 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18147.1.patch, HIVE-18147.2.patch > > > Between the time a new port is picked up to start HMS or HS2, and the time > when HMS or HS2 is started, the port can get used by some other service > resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[
https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267276#comment-16267276
]
Vihang Karajgaonkar commented on HIVE-17528:
{noformat}
./itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:212:
String defaultTestSrcTables =
"src,src1,srcbucket,srcbucket2,src_json,src_thrift,src_sequencefile,srcpart,"
+: warning: Line is longer than 100 characters (found 113).
./itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:213:
"alltypesorc,src_hbase,cbo_t1,cbo_t2,cbo_t3,src_cbo,part,lineitem,alltypesparquet";:
warning:
'"alltypesorc,src_hbase,cbo_t1,cbo_t2,cbo_t3,src_cbo,part,lineitem,alltypesparquet"'
have incorrect indentation level 6, expected level should be 8.
{noformat}
Hi [~Ferd] Can you please fix the yetus errors above? Also, are
parquet_vectorization_div0.q and parquet_vectorization_0 failures related?
> Add more q-tests for Hive-on-Spark with Parquet vectorized reader
> -
>
> Key: HIVE-17528
> URL: https://issues.apache.org/jira/browse/HIVE-17528
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch,
> HIVE-17528.3.patch, HIVE-17528.4.patch, HIVE-17528.5.patch,
> HIVE-17528.7.patch, HIVE-17528.patch
>
>
> Most of the vectorization related q-tests operate on ORC tables using Tez. It
> would be good to add more coverage on a different combination of engine and
> file-format. We can model existing q-tests using parquet tables and run it
> using TestSparkCliDriver
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267269#comment-16267269
]
Hive QA commented on HIVE-16708:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be1f847 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore itests/hive-minikdc U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8004/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
> https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96
>
[jira] [Updated] (HIVE-18124) clean up isAcidTable() API vs isInsertOnlyTable()
[ https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18124: -- Summary: clean up isAcidTable() API vs isInsertOnlyTable() (was: clean up isAcidTable() API) > clean up isAcidTable() API vs isInsertOnlyTable() > -- > > Key: HIVE-18124 > URL: https://issues.apache.org/jira/browse/HIVE-18124 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the > methods in AcidUtils and dependent places are very muddled. > Need to clean it up so that there is a isTransactional(Table) that checks > transactional=true setting and isAcid(Table) to mean full ACID and > isInsertOnly(Table) to mean MM tables. > This would accurately describe the semantics of the tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267199#comment-16267199 ] Vihang Karajgaonkar commented on HIVE-16708: [~thejas] [~aihuaxu] Can you please review the patch? Thanks! > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16708: --- Attachment: HIVE-16708.02.patch Attaching patch for master branch. > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch, HIVE-16708.02.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18150) Upgrade Spark Version to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-18150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-18150: --- > Upgrade Spark Version to 2.2.0 > -- > > Key: HIVE-18150 > URL: https://issues.apache.org/jira/browse/HIVE-18150 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267060#comment-16267060
]
Andrew Sherman commented on HIVE-18136:
---
Thank you [~kgyrtkirk] for the commit.
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Fix For: 3.0.0
>
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267014#comment-16267014
]
Hive QA commented on HIVE-18005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12899417/HIVE-18005.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8003/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8003/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-8003/succeeded/202_UTBatch_service_8_tests,
remoteFile=/home/hiveptest/35.192.234.250-hiveptest-0/logs/,
getExitCode()=255, getException()=null, getUser()=hiveptest,
getHost()=35.192.234.250, getInstance()=0]: 'Warning: Permanently added
'35.192.234.250' (ECDSA) to the list of known hosts.
receiving incremental file list
./
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.TestLdapAtnProviderWithMiniDS.xml
0 0%0.00kB/s0:00:00
92,069 100%1.29MB/s0:00:00 (xfr#1, to-chk=12/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.ldap.TestCustomQueryFilter.xml
0 0%0.00kB/s0:00:00
88,032 100% 842.83kB/s0:00:00 (xfr#2, to-chk=11/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.ldap.TestQuery.xml
0 0%0.00kB/s0:00:00
88,018 100% 834.52kB/s0:00:00 (xfr#3, to-chk=10/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.ldap.TestUserFilter.xml
0 0%0.00kB/s0:00:00
88,002 100% 834.36kB/s0:00:00 (xfr#4, to-chk=9/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.ldap.TestUserSearchFilter.xml
0 0%0.00kB/s0:00:00
88,441 100% 630.42kB/s0:00:00 (xfr#5, to-chk=8/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.cli.TestCLIServiceConnectionLimits.xml
0 0%0.00kB/s0:00:00
89,312 100% 636.63kB/s0:00:00 (xfr#6, to-chk=7/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.cli.TestCLIServiceRestore.xml
0 0%0.00kB/s0:00:00
87,779 100% 621.17kB/s0:00:00 (xfr#7, to-chk=6/14)
TEST-202_UTBatch_service_8_tests-TEST-org.apache.hive.service.cli.TestHiveSQLException.xml
0 0%0.00kB/s0:00:00
88,523 100% 621.93kB/s0:00:00 (xfr#8, to-chk=5/14)
maven-test.txt
0 0%0.00kB/s0:00:00
5,950 100% 41.80kB/s0:00:00 (xfr#9, to-chk=4/14)
logs/
logs/derby.log
0 0%0.00kB/s0:00:00
989 100%6.95kB/s0:00:00 (xfr#10, to-chk=1/14)
logs/hive.log
0 0%0.00kB/s0:00:00
35,127,296 0% 33.50MB/s0:02:02
91,193,344 2% 43.51MB/s0:01:32
148,045,824 3% 47.06MB/s0:01:24
205,291,520 4% 48.95MB/s0:01:20
262,668,288 6% 54.17MB/s0:01:11
320,143,360 7% 54.48MB/s0:01:10
377,585,664 8% 54.60MB/s0:01:08
434,470,912 10% 54.50MB/s0:01:08
491,978,752 11% 54.54MB/s0:01:06
549,322,752 12% 54.53MB/s0:01:05
606,535,680 14% 54.46MB/s0:01:04
663,715,840 15% 54.56MB/s0:01:03
721,551,360 17% 54.58MB/s0:01:02
779,321,344 18% 54.69MB/s0:01:01
836,501,504 19% 54.72MB/s0:01:00
873,431,040 20% 49.78MB/s0:01:05
917,798,912 21% 46.68MB/s0:01:09
966,066,176 22% 44.38MB/s0:01:11
1,017,806,848 24% 43.05MB/s0:01:12
1,071,906,816 25% 47.26MB/s0:01:05
1,127,317,504 26% 49.92MB/s0:01:00
1,183,547,392 27% 51.85MB/s0:00:57
1,239,875,584 29% 52.97MB/s0:00:55
1,296,859,136 30% 53.63MB/s0:00:53
1,343,520,768 31% 51.43MB/s0:00:54
1,401,126,912 33% 51.68MB/s0:00:53
1,458,765,824 34% 51.99MB/s0:00:52
1,516,175,360 35% 52.12MB/s0:00:50
1,573,748,736 37% 54.86MB/s0:00:47
1,621,327,872 38% 52.50MB/s0:00:48
1,666,121,728 39% 49.34MB/s0:00:50
1,717,272,576 40% 47.85MB/s0:00:51
1,764,950,016 41% 45.49MB/s0:00:52
1,819,017,216 42% 46.45MB/s0:00:50
1,867,776,000 44%
[jira] [Commented] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266958#comment-16266958
]
Hive QA commented on HIVE-18005:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
9s{color} | {color:green} storage-api generated 0 new + 2 unchanged - 1 fixed =
2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} storage-api: The patch generated 0 new + 35
unchanged - 4 fixed = 35 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 93 unchanged - 0 fixed
= 94 total (was 93) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be1f847 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8003/yetus/diff-checkstyle-ql.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8003/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18005:
Status: Patch Available (was: Open)
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18005:
Attachment: HIVE-18005.01.patch
#1)
* there were a few other types missing from that estimation...
* this will need some more work later; but at least it's not 0 anymore...this
will break a few qtests.
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18005.01.patch
>
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18005) Improve size estimation for array() to be not 0
[
https://issues.apache.org/jira/browse/HIVE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-18005:
---
Assignee: Zoltan Haindrich
> Improve size estimation for array() to be not 0
> ---
>
> Key: HIVE-18005
> URL: https://issues.apache.org/jira/browse/HIVE-18005
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>
> happens only in case the array is not from a column; and the array contains
> no column references
> {code}
> EXPLAIN
> SELECT sort_array(array("b", "d", "c", "a")),array("1","2") FROM t
> ...
> Statistics: Num rows: 1 Data size: 0 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18141) Fix StatsUtils.combineRange to combine intervals
[
https://issues.apache.org/jira/browse/HIVE-18141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18141:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
pushed to master. thank you Ashutosh for the review!
> Fix StatsUtils.combineRange to combine intervals
>
>
> Key: HIVE-18141
> URL: https://issues.apache.org/jira/browse/HIVE-18141
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 3.0.0
>
> Attachments: HIVE-18141.01.patch, HIVE-18141.02.patch
>
>
> the current [combinedRange
> implementation|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1984]
> in its current form "combines" only ranges which contain eachother
> but the comments suggests that the intention was to capture the case when the
> 2 intervals are overlap; can be checked with the following testcase:
> {code}
> @Test
> public void test11() {
> Range r1 = new Range(0, 1);
> Range r2 = new Range(1, 11);
> Range r3 = StatsUtils.combineRange(r1, r2);
> assertNotNull(r3);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18149: Component/s: Statistics > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266714#comment-16266714 ] Zoltan Haindrich edited comment on HIVE-18149 at 11/27/17 12:22 PM: a longer term improvement would be to estimate a deserialization factor by estimating the "online" rowsize and divide it with an estimated "offline" rowsize... was (Author: kgyrtkirk): possibly an alternative option would be to estimate a deserialization factor by estimating the "online" rowsize and divide it with an estimated "offline" rowsize... > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
