[jira] [Commented] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488443#comment-16488443
]
Hive QA commented on HIVE-17317:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
49s{color} | {color:blue} standalone-metastore in master has 216 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore: The patch generated 10 new + 919
unchanged - 3 fixed = 929 total (was 922) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
7s{color} | {color:red} standalone-metastore generated 1 new + 215 unchanged -
1 fixed = 216 total (was 216) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | Dead store to poolConnFactory in
org.apache.hadoop.hive.metastore.datasource.DbCPDataSourceProvider.create(Configuration)
At
DbCPDataSourceProvider.java:org.apache.hadoop.hive.metastore.datasource.DbCPDataSourceProvider.create(Configuration)
At DbCPDataSourceProvider.java:[line 100] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11173/dev-support/hive-personality.sh
|
| git revision | master / 29e4f3b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11173/yetus/diff-checkstyle-standalone-metastore.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11173/yetus/new-findbugs-standalone-metastore.html
|
| modules | C: common itests/hive-unit standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11173/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make Dbcp configurable using hive properties in hive-site.xml
> -
>
> Key: HIVE-17317
> URL: https://issues.apache.org/jira/browse/HIVE-17317
> Project: Hive
> Issue Type: Sub-task
>
[jira] [Commented] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488442#comment-16488442 ] Prasanth Jayachandran commented on HIVE-19629: -- Thanks [~mmccline] for catching that. Will address it in the next patch. > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[
https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488441#comment-16488441
]
Matt McCline commented on HIVE-19629:
-
The new line 403 in RecordReaderImpl does not look right. To convert from
DECIMAL_64 long to HiveDecimalWritable you should use the
HiveDecimalWritable.deserialize64 method that pass the long and the scale...
{noformat}
result.setFromLong(((Decimal64ColumnVector) vector).vector[row]);{noformat}
> Enable Decimal64 reader after orc version upgrade
> -
>
> Key: HIVE-19629
> URL: https://issues.apache.org/jira/browse/HIVE-19629
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch
>
>
> ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for
> making use of decimal 64 column vectors. Also LLAP IO will need a new reader
> to reader from long stream to decimal 64.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488418#comment-16488418
]
Hive QA commented on HIVE-19557:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924791/HIVE-19557.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14395 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11172/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11172/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924791 - PreCommit-HIVE-Build
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19557.01.patch, HIVE-19557.02.patch,
> HIVE-19557.03.patch, HIVE-19557.04.patch, HIVE-19557.04.patch
>
>
> in StatsRulesProcFactory
> [https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
> are assumed to be an integer]; however this is currently not true - and the
> resulting exception is handled as a default case... for N/3
> {code}
> set hive.explain.user=true;
> create table d1(d date);
> -- tblproperties('transactional'='false');
> insert into d1 values
> ('2010-10-01'),
> ('2010-10-02'),
> ('2010-10-03'),
> ('2010-10-04'),
> ('2010-10-05'),
> ('2010-10-06'),
> ('2010-10-07'),
> ('2010-10-08'),
> ('2010-10-09'),
> ('2010-10-10');
> analyze table d1 compute statistics for columns;
> desc formatted d1;
> desc formatted d1 d;
> explain
> select 'stats: FIL ~0 read',count(1) from d1 where d < '2010-03-01';
> explain
> select 'stats: FIL estimate some read',count(1) from d1 where d <
> '2010-10-03';
> explain
> select 'stats: FIL estimate all read',count(1) from d1 where d < '2010-11-03';
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[ https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488413#comment-16488413 ] Deepak Jaiswal commented on HIVE-18875: --- I would have loved to have a ptest run had it not been ruined by recent changes. I had two positions in the queue and none of them ran and now I am not even in the queue. Its a disaster. > Enable SMB Join by default in Tez > - > > Key: HIVE-18875 > URL: https://issues.apache.org/jira/browse/HIVE-18875 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch, > HIVE-18875.3.patch, HIVE-18875.4.patch, HIVE-18875.5.patch, HIVE-18875.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19559) SparkClientImpl shouldn't name redirector thread "RemoteDriver"
[
https://issues.apache.org/jira/browse/HIVE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488405#comment-16488405
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19559:
-
No test failures.
> SparkClientImpl shouldn't name redirector thread "RemoteDriver"
> ---
>
> Key: HIVE-19559
> URL: https://issues.apache.org/jira/browse/HIVE-19559
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19559.1.patch
>
>
> The thread responsible for re-directing the stdout / stderr of the
> spark-submit process are named {{RemoteDriver...}} which is misleading
> because these threads are not re-directing output from the {{RemoteDriver}},
> just from the spark-submit stdout / stderr.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19557) stats: filters for dates are not taking advantage of min/max values
[
https://issues.apache.org/jira/browse/HIVE-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488399#comment-16488399
]
Hive QA commented on HIVE-19557:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 39 unchanged - 0 fixed
= 40 total (was 39) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 10 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 2322 unchanged - 0 fixed = 2323
total (was 2322) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Comparison of String objects using == or != in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.evaluateComparator(Statistics,
ExprNodeGenericFuncDesc, long) At StatsRulesProcFactory.java:== or != in
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$FilterStatsRule.evaluateComparator(Statistics,
ExprNodeGenericFuncDesc, long) At StatsRulesProcFactory.java:[line 757] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11172/dev-support/hive-personality.sh
|
| git revision | master / 29e4f3b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11172/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11172/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11172/yetus/new-findbugs-ql.html
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11172/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> stats: filters for dates are not taking advantage of min/max values
> ---
>
> Key: HIVE-19557
> URL: https://issues.apache.org/jira/browse/HIVE-19557
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19557.01.patch,
[jira] [Commented] (HIVE-19559) SparkClientImpl shouldn't name redirector thread "RemoteDriver"
[
https://issues.apache.org/jira/browse/HIVE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488378#comment-16488378
]
Hive QA commented on HIVE-19559:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924608/HIVE-19559.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14394 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11171/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11171/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11171/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924608 - PreCommit-HIVE-Build
> SparkClientImpl shouldn't name redirector thread "RemoteDriver"
> ---
>
> Key: HIVE-19559
> URL: https://issues.apache.org/jira/browse/HIVE-19559
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19559.1.patch
>
>
> The thread responsible for re-directing the stdout / stderr of the
> spark-submit process are named {{RemoteDriver...}} which is misleading
> because these threads are not re-directing output from the {{RemoteDriver}},
> just from the spark-submit stdout / stderr.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19559) SparkClientImpl shouldn't name redirector thread "RemoteDriver"
[
https://issues.apache.org/jira/browse/HIVE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488348#comment-16488348
]
Hive QA commented on HIVE-19559:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} spark-client in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11171/dev-support/hive-personality.sh
|
| git revision | master / 29e4f3b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: spark-client U: spark-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11171/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SparkClientImpl shouldn't name redirector thread "RemoteDriver"
> ---
>
> Key: HIVE-19559
> URL: https://issues.apache.org/jira/browse/HIVE-19559
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19559.1.patch
>
>
> The thread responsible for re-directing the stdout / stderr of the
> spark-submit process are named {{RemoteDriver...}} which is misleading
> because these threads are not re-directing output from the {{RemoteDriver}},
> just from the spark-submit stdout / stderr.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488337#comment-16488337 ] Sergey Shelukhin edited comment on HIVE-19690 at 5/24/18 2:53 AM: -- [~jcamachorodriguez] [~ashutoshc] can you take a look? Without the java changes, the last query in the q file produces incorrect results (regardless of Tez, vectorization, CBO, etc.). That is because a list of spray columns (all 3 columns) matches the list of spray+distinct from the other branch, so it creates one RS for two GBY; however, the GBY for v1, v2, v3 has no distinct and is thus not equipped to handle the distinct-specific rows from the RS. was (Author: sershe): [~jcamachorodriguez] [~ashutoshc] can you take a look? Without the java changes, the last query in the q file produces incorrect results (regardless of Tez, vectorization, CBO, etc.). > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19690: Status: Patch Available (was: Open) [~jcamachorodriguez] [~ashutoshc] can you take a look? Without the java changes, the last query in the q file produces incorrect results (regardless of Tez, vectorization, CBO, etc.). > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19690: Attachment: HIVE-19690.patch > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19691) Start SessionState in materialized views registry
[ https://issues.apache.org/jira/browse/HIVE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19691: --- Attachment: HIVE-19691.patch > Start SessionState in materialized views registry > - > > Key: HIVE-19691 > URL: https://issues.apache.org/jira/browse/HIVE-19691 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19691.patch > > > SessionState is not initialized when we load the materialized views, which > leads to a NullPointerException and other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19691) Start SessionState in materialized views registry
[ https://issues.apache.org/jira/browse/HIVE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19691: -- > Start SessionState in materialized views registry > - > > Key: HIVE-19691 > URL: https://issues.apache.org/jira/browse/HIVE-19691 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > SessionState is not initialized when we load the materialized views, which > leads to a NullPointerException and other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19691) Start SessionState in materialized views registry
[ https://issues.apache.org/jira/browse/HIVE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19691: --- Status: Patch Available (was: In Progress) > Start SessionState in materialized views registry > - > > Key: HIVE-19691 > URL: https://issues.apache.org/jira/browse/HIVE-19691 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > SessionState is not initialized when we load the materialized views, which > leads to a NullPointerException and other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19691) Start SessionState in materialized views registry
[ https://issues.apache.org/jira/browse/HIVE-19691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19691 started by Jesus Camacho Rodriguez. -- > Start SessionState in materialized views registry > - > > Key: HIVE-19691 > URL: https://issues.apache.org/jira/browse/HIVE-19691 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > SessionState is not initialized when we load the materialized views, which > leads to a NullPointerException and other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch
[
https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488327#comment-16488327
]
Hive QA commented on HIVE-19660:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924615/HIVE-19660-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 14207 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_2]
(batchId=162)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail]
(batchId=95)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=225)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=229)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=236)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=236)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11170/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11170/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11170/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924615 - PreCommit-HIVE-Build
> update branch-3 to be version 3.1 and fix storage-api mismatch
> --
>
> Key: HIVE-19660
> URL: https://issues.apache.org/jira/browse/HIVE-19660
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19660-branch-3.patch
>
>
> Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is
> marked as 3.0 and master as 3.1-SNAPSHOT.
> Also storage-api features break the build on branch-3 because storage-api
> version does not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19690: --- > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-19615: -- Status: Patch Available (was: Reopened) > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, > HIVE-19615.4.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-19615: -- Attachment: HIVE-19615.4.patch > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, > HIVE-19615.4.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin and analyze: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18866: Resolution: Fixed Assignee: Gopal V (was: Sergey Shelukhin) Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Committed to branches > Semijoin and analyze: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: performance > Fix For: 3.1.0, 4.0.0 > > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.02.patch, HIVE-18866.03.patch, HIVE-18866.04.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488305#comment-16488305
]
slim bouguerra commented on HIVE-19684:
---
added tests case.
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19684.2.patch, HIVE-19684.patch
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Attachment: HIVE-19684.2.patch
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19684.2.patch, HIVE-19684.patch
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch
[ https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19660: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks, committed > update branch-3 to be version 3.1 and fix storage-api mismatch > -- > > Key: HIVE-19660 > URL: https://issues.apache.org/jira/browse/HIVE-19660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19660-branch-3.patch > > > Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is > marked as 3.0 and master as 3.1-SNAPSHOT. > Also storage-api features break the build on branch-3 because storage-api > version does not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19259:
---
Attachment: HIVE-19259.03.patch
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19259.01.patch, HIVE-19259.02.patch,
> HIVE-19259.03.patch, HIVE-19259.patch
>
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as ( select `foo`.`id` from `default`.`foo` ), tmp_2 as (select
> `bar`.`id` from `default`.`bar` ) select `tmp_1`.`id` from tmp_1 union all
> select `tmp_2`.`id` from tmp_2_
> _] used as unionallview at Line 1:14_
> _{code}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch
[ https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488296#comment-16488296 ] Vineet Garg commented on HIVE-19660: [~sershe] Yeah you should push it. We don't need test run for this change. > update branch-3 to be version 3.1 and fix storage-api mismatch > -- > > Key: HIVE-19660 > URL: https://issues.apache.org/jira/browse/HIVE-19660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19660-branch-3.patch > > > Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is > marked as 3.0 and master as 3.1-SNAPSHOT. > Also storage-api features break the build on branch-3 because storage-api > version does not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19644) change WM syntax to avoid conflicts with identifiers starting with a number
[ https://issues.apache.org/jira/browse/HIVE-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19644: Status: Patch Available (was: Open) [~ashutoshc] can you take a look? I tried to change StringLiteral to be either existing generic patterns or the new ones in quotes (after adding quotes to the new ones themselves), but that doesn't get recognized as StringLiteral in select list. So, I just replaced it w/a string literal. > change WM syntax to avoid conflicts with identifiers starting with a number > --- > > Key: HIVE-19644 > URL: https://issues.apache.org/jira/browse/HIVE-19644 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19644.patch > > > Time/etc literals conflict with non-ANSI query column names starting with a > number that were previously supported without quotes (e.g. 30days). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19644) change WM syntax to avoid conflicts with identifiers starting with a number
[ https://issues.apache.org/jira/browse/HIVE-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19644: Attachment: HIVE-19644.patch > change WM syntax to avoid conflicts with identifiers starting with a number > --- > > Key: HIVE-19644 > URL: https://issues.apache.org/jira/browse/HIVE-19644 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19644.patch > > > Time/etc literals conflict with non-ANSI query column names starting with a > number that were previously supported without quotes (e.g. 30days). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19660) update branch-3 to be version 3.1 and fix storage-api mismatch
[
https://issues.apache.org/jira/browse/HIVE-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488285#comment-16488285
]
Hive QA commented on HIVE-19660:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 17s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-11170/patches/PreCommit-HIVE-Build-11170.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11170/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> update branch-3 to be version 3.1 and fix storage-api mismatch
> --
>
> Key: HIVE-19660
> URL: https://issues.apache.org/jira/browse/HIVE-19660
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19660-branch-3.patch
>
>
> Currently we have branch-3.0 presumably for 3.0.X releases, but branch-3 is
> marked as 3.0 and master as 3.1-SNAPSHOT.
> Also storage-api features break the build on branch-3 because storage-api
> version does not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1
[
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488282#comment-16488282
]
Hive QA commented on HIVE-19323:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924606/HIVE-19323.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14394 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testCleanup[Embedded]
(batchId=210)
org.apache.hive.beeline.TestSchemaToolCatalogOps.moveDatabase (batchId=237)
org.apache.hive.beeline.TestSchemaToolCatalogOps.moveTable (batchId=237)
org.apache.hive.beeline.TestSchemaToolCatalogOps.moveTableToNonExistentDb
(batchId=237)
org.apache.hive.beeline.TestSchemaToolCatalogOps.moveTableWithExistingTableOfSameNameAlreadyInTargetDatabase
(batchId=237)
org.apache.hive.beeline.TestSchemaToolCatalogOps.moveTableWithinCatalog
(batchId=237)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11169/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11169/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11169/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924606 - PreCommit-HIVE-Build
> Create metastore SQL install and upgrade scripts for 3.1
>
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.2.patch, HIVE-19323.3.patch, HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade
> scripts for 3.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1
[
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488269#comment-16488269
]
Hive QA commented on HIVE-19323:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 59 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11169/dev-support/hive-personality.sh
|
| git revision | master / 29e4f3b |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11169/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11169/yetus/whitespace-tabs.txt
|
| modules | C: . itests/hive-unit packaging standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11169/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create metastore SQL install and upgrade scripts for 3.1
>
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
> Issue Type: Task
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.2.patch, HIVE-19323.3.patch, HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade
> scripts for 3.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18748) Rename table impacts the ACID behavior as table names are not updated in meta-tables.
[ https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18748: -- Attachment: HIVE-18748.06-branch-3.patch > Rename table impacts the ACID behavior as table names are not updated in > meta-tables. > - > > Key: HIVE-18748 > URL: https://issues.apache.org/jira/browse/HIVE-18748 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, DDL > Attachments: HIVE-18748.01-branch-3.patch, > HIVE-18748.02-branch-3.patch, HIVE-18748.02.patch, > HIVE-18748.03-branch-3.patch, HIVE-18748.03.patch, > HIVE-18748.04-branch-3.patch, HIVE-18748.04.patch, > HIVE-18748.05-branch-3.patch, HIVE-18748.05.patch, > HIVE-18748.06-branch-3.patch > > > ACID implementation uses metatables such as TXN_COMPONENTS, > COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPCTION_QUEUE etc to > manage ACID operations. > Per table write ID implementation (HIVE-18192) introduces couple of > metatables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID to manage write ids > allocated per table. > Now, when we rename any tables, it is necessary to update the corresponding > table names in these metatables as well. Otherwise, ACID table operations > won't work properly. > Since, this change is significant and have other side-effects, we propose to > disable rename tables on ACID tables until a fix is figured out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19598: -- Attachment: HIVE-19598.02.patch > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-19598.02.patch > > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19598: - Assignee: (was: Eugene Koifman) > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Priority: Critical > Attachments: HIVE-19598.02.patch > > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19598: -- Status: Patch Available (was: Open) > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-19598.02.patch > > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19598: - Assignee: Eugene Koifman > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-19598.02.patch > > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488240#comment-16488240
]
Eugene Koifman commented on HIVE-19687:
---
what if partition column is an INTEGER? what will this do?
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488237#comment-16488237
]
Vineet Garg commented on HIVE-19687:
[~ekoifman] Can you take a look?
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488233#comment-16488233
]
Hive QA commented on HIVE-19259:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924591/HIVE-19259.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14395 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_gby2]
(batchId=37)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_1_23]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_skew_1_23]
(batchId=112)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11168/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11168/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11168/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924591 - PreCommit-HIVE-Build
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19259.01.patch, HIVE-19259.02.patch,
> HIVE-19259.patch
>
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as ( select `foo`.`id` from `default`.`foo` ), tmp_2 as (select
> `bar`.`id` from `default`.`bar` ) select `tmp_1`.`id` from tmp_1 union all
> select `tmp_2`.`id` from tmp_2_
> _] used as unionallview at Line 1:14_
> _{code}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19688) Make catalogs updatable
[ https://issues.apache.org/jira/browse/HIVE-19688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-19688: - > Make catalogs updatable > --- > > Key: HIVE-19688 > URL: https://issues.apache.org/jira/browse/HIVE-19688 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > The initial changes for catalogs did not include an ability to alter > catalogs. We need to add that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18685) Add catalogs to metastore
[ https://issues.apache.org/jira/browse/HIVE-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-18685: -- Labels: (was: pull-request-available) > Add catalogs to metastore > - > > Key: HIVE-18685 > URL: https://issues.apache.org/jira/browse/HIVE-18685 > Project: Hive > Issue Type: New Feature > Components: Metastore, Parser, Security, SQL >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HMS Catalog Design Doc.pdf > > > SQL supports two levels of namespaces, called in the spec catalogs and > schemas (with schema being equivalent to Hive's database). I propose to add > the upper level of catalog. The attached design doc covers the use cases, > requirements, and brief discussion of how it will be implemented in a > backwards compatible way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19685) OpenTracing support for HMS
[
https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HIVE-19685:
---
Assignee: Todd Lipcon
Status: Patch Available (was: Open)
Attached patch does the most basic integration here. It instruments the Thrift
protocol so that it propagates traces from clients and creates spans around the
thrift calls themselves.
Using opentracing's JDBC driver you can also get the query traces "for free".
Here's a set of steps you can use to try it out:
1. Build the metastore as normal
2. cd into the bin/ target directory
{code}
mvn dependency:get -Dartifact=io.jaegertracing:jaeger-tracerresolver:0.27.0
-Ddest=lib/
mvn dependency:get -Dartifact=io.jaegertracing:jaeger-core:0.27.0 -Ddest=lib/
mvn dependency:get -Dartifact=io.jaegertracing:jaeger-thrift:0.27.0 -Ddest=lib/
mvn dependency:get -Dartifact=io.opentracing.contrib:opentracing-jdbc:0.0.6
-Ddest=lib/
export JAEGER_SERVICE_NAME=hms
export JAEGER_AGENT_HOST=localhost
export JAEGER_AGENT_PORT=6831
export JAEGER_REPORTER_LOG_SPANS=1
export JAEGER_REPORTER_FLUSH_INTERVAL=1000
export JAEGER_SAMPLER_TYPE=const
export JAEGER_SAMPLER_PARAM=1
docker run -d -e \
COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 9411:9411 \
jaegertracing/all-in-one:latest
./bin/schematool -initSchema -dbType derby
bin/start-metastore \
-hiveconf
'javax.jdo.option.ConnectionURL=jdbc:tracing:derby:;databaseName=metastore_db;create=true'
\
-hiveconf
javax.jdo.option.ConnectionDriverName=io.opentracing.contrib.jdbc.TracingDriver
{code}
If you navigate to http://localhost:16686 you should see the jaeger UI. You can
then run some thrift calls against the HMS and you should see the resulting
traces.
I wasn't sure if it was worth adding explicit new unit tests for this. If you
think so, let me know. There is some kind of MockTracer implementation we could
use for testing.
> OpenTracing support for HMS
> ---
>
> Key: HIVE-19685
> URL: https://issues.apache.org/jira/browse/HIVE-19685
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
> Attachments: hive-19685.patch, trace.png
>
>
> When diagnosing performance of metastore operations it isn't always obvious
> why something took a long time. Using a tracing framework can provide an
> end-to-end view of an operation including time spent in dependent systems (eg
> filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate
> OpenTracing, which is a vendor-neutral tracing API into the HMS server and
> client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HIVE-19685: --- Attachment: hive-19685.patch > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Priority: Major > Attachments: hive-19685.patch, trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-19687:
--
Assignee: Vineet Garg
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19687:
---
Status: Patch Available (was: Open)
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19687:
---
Attachment: HIVE-19687.1.patch
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19687:
---
Attachment: (was: HIVE-19687.1.patch)
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Priority: Major
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19686) schematool --createCatalog option fails when using Oracle as the RDBMS
[
https://issues.apache.org/jira/browse/HIVE-19686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-19686:
--
Status: Patch Available (was: Open)
> schematool --createCatalog option fails when using Oracle as the RDBMS
> ---
>
> Key: HIVE-19686
> URL: https://issues.apache.org/jira/browse/HIVE-19686
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.1
>
>
> Attempts to use the schematool --createCatalog option when the metastore is
> using Oracle result in
> {code:java}
> SQL Error code: 1786
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to add catalog
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:941)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1459)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> Caused by: java.sql.SQLSyntaxErrorException: ORA-01786: FOR UPDATE of this
> query expression is not allowed
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:30)
> at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:762)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
> at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1309)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:422)
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:926)
> ... 7 more
> *** schemaTool failed ***{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19687:
---
Attachment: HIVE-19687.1.patch
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19686) schematool --createCatalog option fails when using Oracle as the RDBMS
[
https://issues.apache.org/jira/browse/HIVE-19686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488209#comment-16488209
]
Alan Gates commented on HIVE-19686:
---
Removing the "select for update" in the query to get the largest catalog id
should fix this. This select for update was just to prevent an error, but is
redundant since the catalog id is the primary of the table and thus no
duplicates will be allowed anyway.
> schematool --createCatalog option fails when using Oracle as the RDBMS
> ---
>
> Key: HIVE-19686
> URL: https://issues.apache.org/jira/browse/HIVE-19686
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.1
>
>
> Attempts to use the schematool --createCatalog option when the metastore is
> using Oracle result in
> {code:java}
> SQL Error code: 1786
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to add catalog
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:941)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1459)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> Caused by: java.sql.SQLSyntaxErrorException: ORA-01786: FOR UPDATE of this
> query expression is not allowed
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:30)
> at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:762)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
> at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1309)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:422)
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:926)
> ... 7 more
> *** schemaTool failed ***{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19686) schematool --createCatalog option fails when using Oracle as the RDBMS
[
https://issues.apache.org/jira/browse/HIVE-19686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates reassigned HIVE-19686:
-
> schematool --createCatalog option fails when using Oracle as the RDBMS
> ---
>
> Key: HIVE-19686
> URL: https://issues.apache.org/jira/browse/HIVE-19686
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.1
>
>
> Attempts to use the schematool --createCatalog option when the metastore is
> using Oracle result in
> {code:java}
> SQL Error code: 1786
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to add catalog
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:941)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1459)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> Caused by: java.sql.SQLSyntaxErrorException: ORA-01786: FOR UPDATE of this
> query expression is not allowed
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:30)
> at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:762)
> at
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)
> at
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
> at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1309)
> at
> oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:422)
> at
> org.apache.hive.beeline.HiveSchemaTool.createCatalog(HiveSchemaTool.java:926)
> ... 7 more
> *** schemaTool failed ***{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19259) Create view on tables having union all fail with "Table not found"
[
https://issues.apache.org/jira/browse/HIVE-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488206#comment-16488206
]
Hive QA commented on HIVE-19259:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11168/dev-support/hive-personality.sh
|
| git revision | master / 29e4f3b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11168/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11168/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create view on tables having union all fail with "Table not found"
> --
>
> Key: HIVE-19259
> URL: https://issues.apache.org/jira/browse/HIVE-19259
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.2.1
> Environment: hive-1.2.1
>
>Reporter: Rajkumar Singh
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19259.01.patch, HIVE-19259.02.patch,
> HIVE-19259.patch
>
>
> create view on table with union work well while "union all" failed with table
> not found, here are the reproduce steps.
> {code}
> _hive> create table foo(id int);_
> _OK_
> _Time taken: 0.401 seconds_
> _hive> create table bar(id int);_
> _OK_
>
> _// view on table union_
> _hive> create view unionview as with tmp_1 as ( select * from foo ), tmp_2 as
> (select * from bar ) select * from tmp_1 union select * from tmp_2;_
> _OK_
> _Time taken: 0.517 seconds_
> _hive> select * from unionview;_
> _OK_
> _Time taken: 5.805 seconds_
>
>
> _// view on union all_
> _hive> create view unionallview as with tmp_1 as ( select * from foo ), tmp_2
> as (select * from bar ) select * from tmp_1 union all select * from tmp_2;_
> _OK_
> _Time taken: 1.535 seconds_
> _hive> select * from unionallview;_
> _FAILED: SemanticException Line 1:134 Table not found 'tmp_1' in definition
> of VIEW unionallview [_
> _with tmp_1 as (
[jira] [Commented] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488190#comment-16488190 ] Prasanth Jayachandran commented on HIVE-19629: -- Will post the patch in RB after ORC upgrade. Patch is up for review though. [~sershe] can you please take a look? > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19629: - Attachment: HIVE-19629.2.patch > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19629) Enable Decimal64 reader after orc version upgrade
[ https://issues.apache.org/jira/browse/HIVE-19629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488186#comment-16488186 ] Prasanth Jayachandran commented on HIVE-19629: -- Found some issues wrt. mixing up orc write version (one with decimal64 and one without). Updated patch and tests. > Enable Decimal64 reader after orc version upgrade > - > > Key: HIVE-19629 > URL: https://issues.apache.org/jira/browse/HIVE-19629 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19629.1.patch, HIVE-19629.2.patch > > > ORC 1.5.0 supports new fast decimal 64 reader. New VRB has to be created for > making use of decimal 64 column vectors. Also LLAP IO will need a new reader > to reader from long stream to decimal 64. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488176#comment-16488176 ] Jesus Camacho Rodriguez commented on HIVE-19615: I have already reverted it. > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster
[
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488177#comment-16488177
]
Hive QA commented on HIVE-19340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924588/HIVE-19340.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14395 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11167/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11167/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11167/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924588 - PreCommit-HIVE-Build
> Disable timeout of transactions opened by replication task at target cluster
>
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Attachments: HIVE-19340.01.patch, HIVE-19340.02.patch
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted
> automatically due to time-out. Aborting of transaction started by replication
> task may leads to inconsistent state at target which needs additional
> overhead to clean-up. So, it is proposed to mark the transactions opened by
> replication task as special ones and shouldn't be aborted if heart beat is
> lost. This helps to ensure all ABORT and COMMIT events will always find the
> corresponding txn at target to operate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488174#comment-16488174 ] slim bouguerra commented on HIVE-19615: --- [~jcamachorodriguez] let me know when is it reverted > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19504:
---
Attachment: HIVE-19504.02.patch
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.01.patch, HIVE-19504.02.patch,
> HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19508) SparkJobMonitor getReport doesn't print stage progress in order
[
https://issues.apache.org/jira/browse/HIVE-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-19508:
Status: Patch Available (was: Open)
Sorting the stages considering them as int values than strings.
> SparkJobMonitor getReport doesn't print stage progress in order
> ---
>
> Key: HIVE-19508
> URL: https://issues.apache.org/jira/browse/HIVE-19508
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19508.1.patch
>
>
> You can end up with a progress output like this:
> {code}
> Stage-10_0: 0/29 Stage-11_0: 0/44Stage-12_0: 0/11
> Stage-13_0: 0/1 Stage-8_0: 258(+76)/468 Stage-9_0: 0/165
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19681) Fix TestVectorIfStatement
[
https://issues.apache.org/jira/browse/HIVE-19681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved HIVE-19681.
Resolution: Invalid
> Fix TestVectorIfStatement
> -
>
> Key: HIVE-19681
> URL: https://issues.apache.org/jira/browse/HIVE-19681
> Project: Hive
> Issue Type: Test
> Components: Vectorization
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> {{TestVectorIfStatement}} generates interesting batches (injection of random
> repeating null column values and repeating non-null values) when evaluating
> the vectorized expressions. But the modification of random rows is done after
> the row mode is evaluated. Hence it is likely that comparison results will
> fail. I am not sure how its working in the first place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19508) SparkJobMonitor getReport doesn't print stage progress in order
[
https://issues.apache.org/jira/browse/HIVE-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-19508:
Attachment: HIVE-19508.1.patch
> SparkJobMonitor getReport doesn't print stage progress in order
> ---
>
> Key: HIVE-19508
> URL: https://issues.apache.org/jira/browse/HIVE-19508
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19508.1.patch
>
>
> You can end up with a progress output like this:
> {code}
> Stage-10_0: 0/29 Stage-11_0: 0/44Stage-12_0: 0/11
> Stage-13_0: 0/1 Stage-8_0: 258(+76)/468 Stage-9_0: 0/165
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[
https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488158#comment-16488158
]
Prasanth Jayachandran commented on HIVE-19685:
--
{quote}if you drop one of the tracer implementations onto the classpath, it's
up to you to configure it and provide your own trace collection infrastructure
{quote}
Sounds good!
Agreed that this will be super useful for dev figuring out performance issues.
Once we have the infrastructure we can possibly extend this end-to-end for
entire lifecycle of the query (better PerfLogger). Looking forward to this.
> OpenTracing support for HMS
> ---
>
> Key: HIVE-19685
> URL: https://issues.apache.org/jira/browse/HIVE-19685
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Reporter: Todd Lipcon
>Priority: Major
> Attachments: trace.png
>
>
> When diagnosing performance of metastore operations it isn't always obvious
> why something took a long time. Using a tracing framework can provide an
> end-to-end view of an operation including time spent in dependent systems (eg
> filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate
> OpenTracing, which is a vendor-neutral tracing API into the HMS server and
> client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[
https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488142#comment-16488142
]
Todd Lipcon commented on HIVE-19685:
[~prasanth_j] sorry, didn't see your question as it came while I was writing
the above comment. I think I partially answered it, but I'll give slightly more
color:
- the default configuration would be a "no-op" tracer, which should have no
measurable overhead.
- if you drop one of the tracer implementations onto the classpath, it's up to
you to configure it and provide your own trace collection infrastructure. In
the case of Jaeger, for example, I've been using a configuration like:
{code}
export JAEGER_SERVICE_NAME=hms
export JAEGER_AGENT_HOST=my-machine.example.com
export JAEGER_AGENT_PORT=6831
export JAEGER_REPORTER_FLUSH_INTERVAL=1000
export JAEGER_SAMPLER_TYPE=const
export JAEGER_SAMPLER_PARAM=1
{code}
And on my-machine.example.com I run some docker images provided by the Jaeger
community. The simplest docker image they provide uses an in-memory store, but
it can also write to Cassandra or Elastic Search as backends. It also provides
the UI as seen in my screenshot.
Personally I've found this very useful to understand HMS performance issues
during development, but I'm not sure if many end-users who deploy Hive would
bother to set it up. IMO that's OK -- we can treat it as a dev-only feature
without adding much maintenance burden.
> OpenTracing support for HMS
> ---
>
> Key: HIVE-19685
> URL: https://issues.apache.org/jira/browse/HIVE-19685
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Reporter: Todd Lipcon
>Priority: Major
> Attachments: trace.png
>
>
> When diagnosing performance of metastore operations it isn't always obvious
> why something took a long time. Using a tracing framework can provide an
> end-to-end view of an operation including time spent in dependent systems (eg
> filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate
> OpenTracing, which is a vendor-neutral tracing API into the HMS server and
> client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[
https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488139#comment-16488139
]
Todd Lipcon commented on HIVE-19685:
Before working on a patch against trunk, wanted to run a summary of the design
by folks:
- add POM dependencies from the metastore module to opentracing
-- the "opentracing" project itself is just APIs, not coupled to any tracing
implementation. You can think of it like Slf4j where the user has to provide an
implementation of their choice. opentracing supports a number of
implementations, the most popular being Jaeger (from Uber) and Zipkin (from
Twitter) as well as a commercial implementation provided by LightStep.
-- we'd also include the 'tracerresolver' module. This uses a Java
ServiceLoader to look for appropriate plugins on the classpath at start time.
This would allow a user to drop Jaeger or Zipkin onto the classpath and enable
tracing without recompilation. The tracing implementation's configuration is
implementation-specific. For example, Jaeger's configuration is by environment
variables.
- add a POM dependency to opentracing-thrift, which is some simple utility code
to wrap a TProtocol and TProcessor so that the client and server propagate a
trace context between them. This allows a trace to be correlated between two
processes (eg HS2 and HMS). We might want to shade these classes since they'd
show up in consumer classpaths who are using the HMS client.
In order to get the tracing of JDBC calls as shown in the screenshot above, no
code is necessary. The user just adds the opentracing-jdbc jar to their
classpath and then appropriately configures their JDBC connection string. It
acts like a "passthrough" to the underlying JDBC driver.
The above is the basic integration. Beyond that, we can add small bits of
instrumentation to interesting points of the code. For example:
{code}
private boolean ensureDbInit() {
try (Scope s =
GlobalTracer.get().buildSpan("MetaStoreDirectSQL.ensureDbInit")
.startActive(true)) {
guts of method ...
+}
}
{code}
this makes it easy to spot issues like HIVE-19310.
Thoughts?
> OpenTracing support for HMS
> ---
>
> Key: HIVE-19685
> URL: https://issues.apache.org/jira/browse/HIVE-19685
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Reporter: Todd Lipcon
>Priority: Major
> Attachments: trace.png
>
>
> When diagnosing performance of metastore operations it isn't always obvious
> why something took a long time. Using a tracing framework can provide an
> end-to-end view of an operation including time spent in dependent systems (eg
> filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate
> OpenTracing, which is a vendor-neutral tracing API into the HMS server and
> client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[
https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488128#comment-16488128
]
Jesus Camacho Rodriguez edited comment on HIVE-19615 at 5/23/18 10:37 PM:
--
I tested in master:
{code}
?Exception
explain select (cstring1 is null ) AS is_null, (cint is not null ) as isnotnull
FROM druid_table" fname=druidmini_test1.q
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/
for specific test cases logs.
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2229)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:180)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestMiniDruidCliDriver.testCliDriver:59 Client execution failed with
error code = 10001 running "
explain select (cstring1 is null ) AS is_null, (cint is not null ) as isnotnull
FROM druid_table" fname=druidmini_test1.q
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/
for specific test cases logs.
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 03:01 min
[INFO] Finished at: 2018-05-23T15:12:55-07:00
[INFO] Final Memory: 81M/1478M
[INFO]
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488136#comment-16488136 ] Prasanth Jayachandran commented on HIVE-19685: -- Where does trace data gets stored? Is traces sink/storage configurable via properties file or does it need compile time dependencies to external system (like cassandra)? > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Priority: Major > Attachments: trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster
[
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488135#comment-16488135
]
Hive QA commented on HIVE-19340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
57s{color} | {color:blue} standalone-metastore in master has 216 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore: The patch generated 1 new + 551
unchanged - 1 fixed = 552 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11167/dev-support/hive-personality.sh
|
| git revision | master / 7eadc07 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11167/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11167/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable timeout of transactions opened by replication task at target cluster
>
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Attachments: HIVE-19340.01.patch, HIVE-19340.02.patch
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted
> automatically due to time-out. Aborting of transaction started by replication
> task may leads to inconsistent state at target which needs additional
> overhead to clean-up. So, it is proposed to mark the transactions opened by
> replication task as special ones and shouldn't be aborted if heart beat is
> lost.
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[
https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488134#comment-16488134
]
Jesus Camacho Rodriguez commented on HIVE-19615:
[~bslim], failure is due to renaming of tables in q test files, {{druid_table}}
no longer exists.
> Proper handling of is null and not is null predicate when pushed to Druid
> -
>
> Key: HIVE-19615
> URL: https://issues.apache.org/jira/browse/HIVE-19615
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch
>
>
> Recent development in Druid introduced new semantic of null handling
> [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346]
> Based on those changes when need to honer push down of expressions with is
> null/ is not null predicates.
> The prosed fix overrides the mapping of Calcite Function to Druid Expression
> to much the correct semantic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[
https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488134#comment-16488134
]
Jesus Camacho Rodriguez edited comment on HIVE-19615 at 5/23/18 10:36 PM:
--
[~bslim], failure is due to renaming of tables in q test files, {{druid_table}}
no longer exists, you can use master to reproduce it locally.
was (Author: jcamachorodriguez):
[~bslim], failure is due to renaming of tables in q test files, {{druid_table}}
no longer exists.
> Proper handling of is null and not is null predicate when pushed to Druid
> -
>
> Key: HIVE-19615
> URL: https://issues.apache.org/jira/browse/HIVE-19615
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch
>
>
> Recent development in Druid introduced new semantic of null handling
> [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346]
> Based on those changes when need to honer push down of expressions with is
> null/ is not null predicates.
> The prosed fix overrides the mapping of Calcite Function to Druid Expression
> to much the correct semantic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488129#comment-16488129 ] Todd Lipcon commented on HIVE-19685: Following is a screenshot of the trace view for the simple integration I've been playing with: !trace.png! This makes it very easy to find issues like HIVE-19605 since the slow queries stick out like a sore thumb. It's also helpful for clients of the HMS who have also integrated opentracing -- it's easier to see how much of the total operation time of a slow request can be attributed to the HMS vs other contributing factors. > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Priority: Major > Attachments: trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[
https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488128#comment-16488128
]
Jesus Camacho Rodriguez commented on HIVE-19615:
I tested in master:
{code}
?Exception
explain select (cstring1 is null ) AS is_null, (cint is not null ) as isnotnull
FROM druid_table" fname=druidmini_test1.q
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/
for specific test cases logs.
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2229)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:180)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestMiniDruidCliDriver.testCliDriver:59 Client execution failed with
error code = 10001 running "
explain select (cstring1 is null ) AS is_null, (cint is not null ) as isnotnull
FROM druid_table" fname=druidmini_test1.q
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/
for specific test cases logs.
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 03:01 min
[INFO] Finished at: 2018-05-23T15:12:55-07:00
[INFO] Final Memory: 81M/1478M
[INFO]
[ERROR] Failed
[jira] [Updated] (HIVE-19685) OpenTracing support for HMS
[ https://issues.apache.org/jira/browse/HIVE-19685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HIVE-19685: --- Attachment: trace.png > OpenTracing support for HMS > --- > > Key: HIVE-19685 > URL: https://issues.apache.org/jira/browse/HIVE-19685 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Todd Lipcon >Priority: Major > Attachments: trace.png > > > When diagnosing performance of metastore operations it isn't always obvious > why something took a long time. Using a tracing framework can provide an > end-to-end view of an operation including time spent in dependent systems (eg > filesystem operations, RDBMS queries, etc). This JIRA proposes to integrate > OpenTracing, which is a vendor-neutral tracing API into the HMS server and > client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Status: Patch Available (was: Open)
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19684.patch
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Attachment: HIVE-19684.patch
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19684.patch
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488119#comment-16488119 ] slim bouguerra commented on HIVE-19615: --- [~jcamachorodriguez] which branch are you using to reproduce the Druid failure ? i can not reproduced it locally and that test did not fail prior merging this. > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488115#comment-16488115 ] Gunther Hagleitner commented on HIVE-19615: --- [~bslim] see vote on hive dev mailing list. "Unrelated" is no longer good enough for commit. > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488112#comment-16488112 ] slim bouguerra commented on HIVE-19615: --- I don't recall exactly but I have looked at the errors and seems unrelated, will let [~ashutoshc] revert if needed. > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[
https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488110#comment-16488110
]
Jesus Camacho Rodriguez commented on HIVE-19615:
I will revert this patch, also it seems to be causing new issues. Last run:
{code}
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924590/HIVE-19504.01.patch
ERROR: -1 due to no test(s) being added or modified.
ERROR: -1 due to 10 failed/errored test(s), 14317 tests executed
Failed tests:
TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestHiveCli - did not produce a TEST-*.xml file (likely timed out) (batchId=192)
TestHiveSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestIncrementalRows - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestShutdownHook - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestTableOutputFormat - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=254)
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11166/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11166/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11166/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
This message is automatically generated.
ATTACHMENT ID: 12924590 - PreCommit-HIVE-Build
{code}
I can reproduce the Druid failure locally.
> Proper handling of is null and not is null predicate when pushed to Druid
> -
>
> Key: HIVE-19615
> URL: https://issues.apache.org/jira/browse/HIVE-19615
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch
>
>
> Recent development in Druid introduced new semantic of null handling
> [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346]
> Based on those changes when need to honer push down of expressions with is
> null/ is not null predicates.
> The prosed fix overrides the mapping of Calcite Function to Druid Expression
> to much the correct semantic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reopened HIVE-19615: > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19454) Test failure : org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails with java.lang.AssertionError
[
https://issues.apache.org/jira/browse/HIVE-19454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19454:
---
Fix Version/s: 4.0.0
3.1.0
> Test failure :
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 fails
> with java.lang.AssertionError
> --
>
> Key: HIVE-19454
> URL: https://issues.apache.org/jira/browse/HIVE-19454
> Project: Hive
> Issue Type: Bug
> Environment: uname -a
> Linux pts00433-vm40 3.10.0-862.el7.ppc64le #1 SMP Wed Mar 21 22:20:19 UTC
> 2018 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19454.1.patch, HIVE-19454.2.patch
>
>
> org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
> refers to the same test case.
> Maven command used is : mvn -Dtest=TestTxnCommands2 test
> Error:
> {code:java}
> [INFO] Running org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3, Time elapsed:
> 618.215 s <<< FAILURE! - in org.apache.hadoop.hive.ql.TestTxnCommands2
> [ERROR]
> testNonAcidToAcidConversion1(org.apache.hadoop.hive.ql.TestTxnCommands2)
> Time elapsed: 17.557 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1(TestTxnCommands2.java:499)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR] TestTxnCommands2.testNonAcidToAcidConversion1:499
> [INFO]
> [ERROR] Tests run: 44, Failures: 1, Errors: 0, Skipped: 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19615) Proper handling of is null and not is null predicate when pushed to Druid
[ https://issues.apache.org/jira/browse/HIVE-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488107#comment-16488107 ] Gunther Hagleitner commented on HIVE-19615: --- [~ashutoshc]/[~bslim] I do not see a clean run here. Commit by mistake? Can you revert? > Proper handling of is null and not is null predicate when pushed to Druid > - > > Key: HIVE-19615 > URL: https://issues.apache.org/jira/browse/HIVE-19615 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19615.2.patch, HIVE-19615.3.patch, HIVE-19615.patch > > > Recent development in Druid introduced new semantic of null handling > [here|https://github.com/b-slim/druid/commit/219e77aeac9b07dc20dd9ab2dd537f3f17498346] > Based on those changes when need to honer push down of expressions with is > null/ is not null predicates. > The prosed fix overrides the mapping of Calcite Function to Druid Expression > to much the correct semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[ https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488099#comment-16488099 ] Gunther Hagleitner commented on HIVE-18875: --- Ok. Good point about the inputObjInspector this has to be 0. Good catch on the subquery_notin.q I'm confused about what you're saying wrt the join operator close logic. Can you just always execute the close for all parents (regardless of whether they are Dummy or Group by operators? I think checking op traits to see if it's a quick fix might be better than putting it off. > Enable SMB Join by default in Tez > - > > Key: HIVE-18875 > URL: https://issues.apache.org/jira/browse/HIVE-18875 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch, > HIVE-18875.3.patch, HIVE-18875.4.patch, HIVE-18875.5.patch, HIVE-18875.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488093#comment-16488093
]
Hive QA commented on HIVE-19504:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924590/HIVE-19504.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14317 tests
executed
*Failed tests:*
{noformat}
TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestHiveCli - did not produce a TEST-*.xml file (likely timed out) (batchId=192)
TestHiveSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestIncrementalRows - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestShutdownHook - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestTableOutputFormat - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11166/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11166/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11166/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924590 - PreCommit-HIVE-Build
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.01.patch, HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Priority: Blocker (was: Major)
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Fix Version/s: 3.0.0
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19684:
--
Component/s: Physical Optimizer
Druid integration
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
> Components: Druid integration, Physical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19684) Hive stats optimizer wrongly uses stats against non native tables
[
https://issues.apache.org/jira/browse/HIVE-19684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-19684:
-
> Hive stats optimizer wrongly uses stats against non native tables
> -
>
> Key: HIVE-19684
> URL: https://issues.apache.org/jira/browse/HIVE-19684
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Stats of non native tables are inaccurate, thus queries over non native
> tables can not optimized by stats optimizer.
> Take example of query
> {code}
> Explain select count(*) from (select `__time` from druid_test_table limit 1)
> as src ;
> {code}
> the plan will be reduced to
> {code}
> POSTHOOK: query: explain extended select count(*) from (select `__time` from
> druid_test_table limit 1) as src
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Patch Available (was: In Progress) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Attachment: HIVE-19305.1-branch-3.patch > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19683) Flaky tests output TestTezPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-19683:
-
Component/s: Tests
> Flaky tests output TestTezPerfCliDriver
> ---
>
> Key: HIVE-19683
> URL: https://issues.apache.org/jira/browse/HIVE-19683
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
>
> After setting {{hive.optimize.index.filter}} to true,
> {{TestTezPerfCliDriver}} has different results every run for more than 60 q
> files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19683) Flaky tests output TestTezPerfCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko reassigned HIVE-19683:
> Flaky tests output TestTezPerfCliDriver
> ---
>
> Key: HIVE-19683
> URL: https://issues.apache.org/jira/browse/HIVE-19683
> Project: Hive
> Issue Type: Bug
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
>
> After setting {{hive.optimize.index.filter}} to true,
> {{TestTezPerfCliDriver}} has different results every run for more than 60 q
> files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19464) Upgrade Parquet to 1.10.0
[
https://issues.apache.org/jira/browse/HIVE-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488065#comment-16488065
]
Jesus Camacho Rodriguez commented on HIVE-19464:
Uploaded new patch regenerating {{parquet_vectorization_0.q}} for Spark.
> Upgrade Parquet to 1.10.0
> -
>
> Key: HIVE-19464
> URL: https://issues.apache.org/jira/browse/HIVE-19464
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19464.01.patch, HIVE-19464.02.patch,
> HIVE-19464.03.patch, HIVE-19464.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19464) Upgrade Parquet to 1.10.0
[
https://issues.apache.org/jira/browse/HIVE-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488064#comment-16488064
]
Jesus Camacho Rodriguez commented on HIVE-19464:
Uploaded new patch regenerating {{parquet_vectorization_0.q}} for Spark.
> Upgrade Parquet to 1.10.0
> -
>
> Key: HIVE-19464
> URL: https://issues.apache.org/jira/browse/HIVE-19464
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19464.01.patch, HIVE-19464.02.patch,
> HIVE-19464.03.patch, HIVE-19464.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-19464) Upgrade Parquet to 1.10.0
[
https://issues.apache.org/jira/browse/HIVE-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19464:
---
Comment: was deleted
(was: Uploaded new patch regenerating {{parquet_vectorization_0.q}} for Spark.)
> Upgrade Parquet to 1.10.0
> -
>
> Key: HIVE-19464
> URL: https://issues.apache.org/jira/browse/HIVE-19464
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19464.01.patch, HIVE-19464.02.patch,
> HIVE-19464.03.patch, HIVE-19464.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19464) Upgrade Parquet to 1.10.0
[ https://issues.apache.org/jira/browse/HIVE-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-19464: --- Attachment: HIVE-19464.04.patch > Upgrade Parquet to 1.10.0 > - > > Key: HIVE-19464 > URL: https://issues.apache.org/jira/browse/HIVE-19464 > Project: Hive > Issue Type: Improvement > Components: File Formats >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19464.01.patch, HIVE-19464.02.patch, > HIVE-19464.03.patch, HIVE-19464.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[ https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488047#comment-16488047 ] Deepak Jaiswal commented on HIVE-18875: --- [~hagleitn] thanks for looking into the patch. your questions are in italics. _The join operator should not have to check whether the parent is a group by - that seems brittle. Can we always force the logic to close other branches to flush out remaining records? If we introduce any other blocking operators the same logic has to apply, right?_ I agree it is brittle. The logic was put there to establish it is on reducer side so that we dont execute it otherwise. I think it will be simpler to put a flag somewhere or use some other existing info to establish the same info. _Not using the tag in the group by operator (hard code to 0) seems wrong, why is that the correct thing to do?_ The tag is irrelevant in GBY. There is no other use case of tag other than SMB as of now. There is always exactly one OI and SMB may send tag 1 or larger which causes ArrayIndexOutOfBoundsExcpetion. _Why are you turning sortmerge join conversion off explicitly in some test files? Can you explain + add comment there?_ Most of those tests are already testing SMB. The way I structured those tests is such that it first runs the query without SMB and then with SMB, however, since it is now on by default it needs to be explicitly turned off for those sections. subquery_notin.q had typo in it which I fixed after discussing it with [~vgarg]. The query which ran and explain before it were different. _If I read this right, then you're new check in ConvertJoinMapjoin basically makes sure that there is no projection in between gby and join that would alter bucketing or sorting. That is exactly what op traits are for - why can't we use that in this case?_ That would be ideal case. It looks like op traits in its current form are not sufficient to handle all SMB cases. Maybe I can do it as a follow through? > Enable SMB Join by default in Tez > - > > Key: HIVE-18875 > URL: https://issues.apache.org/jira/browse/HIVE-18875 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch, > HIVE-18875.3.patch, HIVE-18875.4.patch, HIVE-18875.5.patch, HIVE-18875.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19504) Change default value for hive.auto.convert.join.shuffle.max.size property
[
https://issues.apache.org/jira/browse/HIVE-19504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488040#comment-16488040
]
Hive QA commented on HIVE-19504:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11166/dev-support/hive-personality.sh
|
| git revision | master / 7eadc07 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11166/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change default value for hive.auto.convert.join.shuffle.max.size property
> -
>
> Key: HIVE-19504
> URL: https://issues.apache.org/jira/browse/HIVE-19504
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19504.01.patch, HIVE-19504.patch
>
>
> The property default value is too low by mistake (10MB), it is missing three
> trailing zeros.
> {code}
> HIVECONVERTJOINMAXSHUFFLESIZE("hive.auto.convert.join.shuffle.max.size",
> 1000L,
>"If hive.auto.convert.join.noconditionaltask is off, this parameter
> does not take affect. \n" +
>"However, if it is on, and the predicted size of the larger input for
> a given join is greater \n" +
>"than this number, the join will not be converted to a dynamically
> partitioned hash join. \n" +
>"The value \"-1\" means no limit."),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19464) Upgrade Parquet to 1.10.0
[
https://issues.apache.org/jira/browse/HIVE-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488010#comment-16488010
]
Hive QA commented on HIVE-19464:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924582/HIVE-19464.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14394 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=115)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11165/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11165/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11165/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924582 - PreCommit-HIVE-Build
> Upgrade Parquet to 1.10.0
> -
>
> Key: HIVE-19464
> URL: https://issues.apache.org/jira/browse/HIVE-19464
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19464.01.patch, HIVE-19464.02.patch,
> HIVE-19464.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[ https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487990#comment-16487990 ] Gunther Hagleitner commented on HIVE-18875: --- If I read this right, then you're new check in ConvertJoinMapjoin basically makes sure that there is no projection in between gby and join that would alter bucketing or sorting. That is exactly what op traits are for - why can't we use that in this case? > Enable SMB Join by default in Tez > - > > Key: HIVE-18875 > URL: https://issues.apache.org/jira/browse/HIVE-18875 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch, > HIVE-18875.3.patch, HIVE-18875.4.patch, HIVE-18875.5.patch, HIVE-18875.6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
