[jira] [Commented] (HIVE-22225) add owner name for create database if missing
[
https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934939#comment-16934939
]
Hive QA commented on HIVE-5:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18675/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18675/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add owner name for create database if missing
> -
>
> Key: HIVE-5
> URL: https://issues.apache.org/jira/browse/HIVE-5
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Sam An
>Assignee: Sam An
>Priority: Minor
> Attachments: HIVE-5.1.patch
>
>
> When Spark connects to HMS, the database owner name is not filled. This could
> happen to other clients as well. We shall fill this in HMS. Ownership info is
> useful for authorizer component in Ranger, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned HIVE-9: -- Assignee: (was: Yuming Wang) > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[
https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934917#comment-16934917
]
Hive QA commented on HIVE-2:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980926/HIVE-2.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18674/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18674/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18674/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980926 - PreCommit-HIVE-Build
> Clean up the error handling in Driver - get rid of global variables
> ---
>
> Key: HIVE-2
> URL: https://issues.apache.org/jira/browse/HIVE-2
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-2.01.patch
>
>
> The error handling in Hive is done with some global variables for no apparent
> reason, as all the data that is gathered to described an exception are
> produced and used at the point where the exception occurred. Thus having
> global variables is misleading.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[
https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934906#comment-16934906
]
Hive QA commented on HIVE-2:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 87 unchanged - 46
fixed = 92 total (was 133) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18674/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18674/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18674/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up the error handling in Driver - get rid of global variables
> ---
>
> Key: HIVE-2
> URL: https://issues.apache.org/jira/browse/HIVE-2
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-2.01.patch
>
>
> The error handling in Hive is done with some global variables for no apparent
> reason, as all the data that is gathered to described an exception are
> produced and used at the point where the exception occurred. Thus having
> global variables is misleading.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934902#comment-16934902 ] Yuming Wang commented on HIVE-9: [* DDL – Alter Database|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterDatabase] Please note, Fix Version/s should include 2.3.7 if we backport HIVE-8472 to branch-2.3. > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned HIVE-9: -- > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934899#comment-16934899
]
Hive QA commented on HIVE-21975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980924/HIVE-21975.1.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=282)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18673/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18673/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980924 - PreCommit-HIVE-Build
> Fix incremental compilation
> ---
>
> Key: HIVE-21975
> URL: https://issues.apache.org/jira/browse/HIVE-21975
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Steve Carlin
>Priority: Major
> Attachments: HIVE-21975.1.patch, HIVE-21975.patch
>
>
> we have an incremental compilation issue around SA ; mostly because of "?
> extends Serializable"
> it could be reproduced with:
> {code}
> git clean -dfx
> mvn install -pl ql -am -DskipTests
> touch `find . -name Sema*A*java` `find . -name Task*Factory.java`
> mvn install -pl ql -DskipTests
> {code}
> error is:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60]
> incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to
> java.util.List>
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49]
> incompatible types: java.util.List>
> cannot be converted to java.util.List extends java.io.Serializable>>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-1: - Attachment: HIVE-1.2.patch > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-1.1.patch, HIVE-1.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934896#comment-16934896
]
Hive QA commented on HIVE-21975:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/util in master has 51 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 188 new + 1675 unchanged - 95
fixed = 1863 total (was 1770) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 2 new + 54 unchanged - 1
fixed = 56 total (was 55) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/core: The patch generated 4 new + 43
unchanged - 0 fixed = 47 total (was 43) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 586
unchanged - 1 fixed = 590 total (was 587) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/util: The patch generated 4 new + 14 unchanged
- 0 fixed = 18 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
11s{color} | {color:red} ql generated 2 new + 1571 unchanged - 0 fixed = 1573
total (was 1571) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Inconsistent synchronization of
org.apache.hadoop.hive.ql.QueryDisplay$TaskDisplay.beginTime; locked 57% of
time Unsynchronized access at QueryDisplay.java:57% of time Unsynchronized
access at QueryDisplay.java:[line 243] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.ql.QueryDisplay$TaskDisplay.endTime; locked 50% of time
Unsynchronized access at QueryDisplay.java:50% of time Unsynchronized access
at QueryDisplay.java:[line 248] |
\\
\\
|| Subsystem ||
[jira] [Commented] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934885#comment-16934885
]
Hive QA commented on HIVE-1:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980909/HIVE-1.1.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16840 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions]
(batchId=81)
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980909 - PreCommit-HIVE-Build
> Llap external client - Need to reduce LlapBaseInputFormat#getSplits()
> footprint
> -
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
> Issue Type: Bug
> Components: llap, UDF
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-1.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While querying through llap external client, LlapBaseInputFormat#getSplits()
> invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods.
> GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies
> around 90% of the split size.
> Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb
> with planBytes[] being common to all the splits and occupying more than 850
> kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size.
> This can be resolved by separating out common parts from actual splits and
> reassembling them at client side.
> We can also provide an option where client can say it does not want to
> reassemble them and can take the control of reassembling in it's hands.
> Splits can be broken like:
> 1) schema split
> 2) plan split
> 3) actual split 1
> 4) actual split 2and so on.
> This greatly reduces the memory(in my case from 5GB(~5000 splits) to around
> 15MB) on server side and hence the data transfer. And this eliminates OOM on
> HS2 side.
> cc [~jdere] [~sankarh] [~thejas]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Patch Available (was: Open) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: HIVE-22197.05.patch > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Open (was: Patch Available) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: (was: HIVE-22197.05.patch) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934875#comment-16934875
]
Hive QA commented on HIVE-1:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-ext-client in master has 1 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 49 new + 96 unchanged - 4
fixed = 145 total (was 100) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-ext-client: The patch generated 3 new + 36
unchanged - 2 fixed = 39 total (was 38) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 53
unchanged - 5 fixed = 62 total (was 58) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} llap-client generated 1 new + 26 unchanged - 0 fixed =
27 total (was 26) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
12s{color} | {color:red} ql generated 1 new + 1570 unchanged - 1 fixed = 1571
total (was 1571) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} llap-ext-client generated 1 new + 1 unchanged - 0
fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-client |
| | org.apache.hadoop.hive.llap.LlapInputSplit.setPlanBytes(byte[]) may
expose internal representation by storing an externally mutable object into
LlapInputSplit.planBytes At LlapInputSplit.java:by storing an externally
mutable object into LlapInputSplit.planBytes At LlapInputSplit.java:[line 95] |
| FindBugs | module:ql |
| | Redundant nullcheck of driverCleanup, which is known to be non-null in
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(String,
ApplicationId) Redundant null check at GenericUDTFGetSplits.java:is known to
be non-null in
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(String,
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934863#comment-16934863
]
Hive QA commented on HIVE-22209:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980896/HIVE-22209.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18671/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18671/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980896 - PreCommit-HIVE-Build
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.1.patch, HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934851#comment-16934851
]
Hive QA commented on HIVE-22209:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18671/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18671/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.1.patch, HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22211) Change maven phase to generate test sources
[
https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934840#comment-16934840
]
Hive QA commented on HIVE-22211:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980890/HIVE-22211.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18670/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18670/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980890 - PreCommit-HIVE-Build
> Change maven phase to generate test sources
> ---
>
> Key: HIVE-22211
> URL: https://issues.apache.org/jira/browse/HIVE-22211
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch,
> HIVE-22211.01.patch
>
>
> Some protobuf files are generated in the wrong phase; so I get compile errors
> because they are not there for eclipse...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[
https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934824#comment-16934824
]
Hive QA commented on HIVE-22205:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
29s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 51 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch llap-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 2
fixed = 6 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} The patch llap-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch kafka-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green}
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[
https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-7:
---
Status: Patch Available (was: Open)
> Tez bucket pruning produces wrong result with shared work optimization
> --
>
> Key: HIVE-7
> URL: https://issues.apache.org/jira/browse/HIVE-7
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-7.1.patch
>
>
> *Reproducer*
> {code:sql}
> set hive.tez.bucket.pruning=true;
> set hive.optimize.shared.work=true;
> CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds
> string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
> CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE;
> CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned
> by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE;
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> set hive.optimize.bucketingsorting=false;
> insert overwrite table tab_part_n10 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_part_n17;
> CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE;
> insert overwrite table tab_n9 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_n16;
> select * from
> (select * from tab_n9 where tab_n9.key = 0)a
> join
> (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join
> tab_part_n10 c on a.key = b.key and b.key = c.key
> order by 1,2,3,4,5,6,7,8,9;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[
https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-7:
---
Attachment: HIVE-7.1.patch
> Tez bucket pruning produces wrong result with shared work optimization
> --
>
> Key: HIVE-7
> URL: https://issues.apache.org/jira/browse/HIVE-7
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-7.1.patch
>
>
> *Reproducer*
> {code:sql}
> set hive.tez.bucket.pruning=true;
> set hive.optimize.shared.work=true;
> CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds
> string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
> CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE;
> CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned
> by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE;
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> set hive.optimize.bucketingsorting=false;
> insert overwrite table tab_part_n10 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_part_n17;
> CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE;
> insert overwrite table tab_n9 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_n16;
> select * from
> (select * from tab_n9 where tab_n9.key = 0)a
> join
> (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join
> tab_part_n10 c on a.key = b.key and b.key = c.key
> order by 1,2,3,4,5,6,7,8,9;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22211) Change maven phase to generate test sources
[
https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934808#comment-16934808
]
Hive QA commented on HIVE-22211:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18670/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18670/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change maven phase to generate test sources
> ---
>
> Key: HIVE-22211
> URL: https://issues.apache.org/jira/browse/HIVE-22211
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch,
> HIVE-22211.01.patch
>
>
> Some protobuf files are generated in the wrong phase; so I get compile errors
> because they are not there for eclipse...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-8: -- Attachment: HIVE-8.01.patch > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-8.01.patch > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-8: -- Status: Patch Available (was: Open) > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-8.01.patch > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[
https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934794#comment-16934794
]
Hive QA commented on HIVE-22205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980849/HIVE-22205.4.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16828 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=289)
org.apache.hadoop.hive.ql.TestTxnCommands.testParallelInsertAnalyzeStats
(batchId=351)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18669/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18669/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18669/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980849 - PreCommit-HIVE-Build
> Upgrade zookeeper and curator versions
> --
>
> Key: HIVE-22205
> URL: https://issues.apache.org/jira/browse/HIVE-22205
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch,
> HIVE-22205.4.patch, HIVE-22205.patch
>
>
> Other components like hadoop have switched to using new ZK versions. So these
> jars end up in classpath for hive services and could cause issues due to
> in-compatible curator versions that hive uses.
> So it makes sense for hive to upgrade the ZK and curator versions to try to
> keep up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-8: - > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[
https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-7:
---
Description:
*Reproducer*
{code:sql}
set hive.tez.bucket.pruning=true;
set hive.optimize.shared.work=true;
CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds
string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING)
CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE;
CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned by
(ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE;
load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
srcbucket_mapjoin_n16 partition(ds='2008-04-08');
load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE
srcbucket_mapjoin_n16 partition(ds='2008-04-08');
load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE
srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE
srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
set hive.optimize.bucketingsorting=false;
insert overwrite table tab_part_n10 partition (ds='2008-04-08')
select key,value from srcbucket_mapjoin_part_n17;
CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING) CLUSTERED
BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE;
insert overwrite table tab_n9 partition (ds='2008-04-08')
select key,value from srcbucket_mapjoin_n16;
select * from
(select * from tab_n9 where tab_n9.key = 0)a
join
(select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join
tab_part_n10 c on a.key = b.key and b.key = c.key
order by 1,2,3,4,5,6,7,8,9;
{code}
> Tez bucket pruning produces wrong result with shared work optimization
> --
>
> Key: HIVE-7
> URL: https://issues.apache.org/jira/browse/HIVE-7
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> *Reproducer*
> {code:sql}
> set hive.tez.bucket.pruning=true;
> set hive.optimize.shared.work=true;
> CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds
> string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
> CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE;
> CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned
> by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE;
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE
> srcbucket_mapjoin_n16 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE
> srcbucket_mapjoin_part_n17 partition(ds='2008-04-08');
> set hive.optimize.bucketingsorting=false;
> insert overwrite table tab_part_n10 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_part_n17;
> CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE;
> insert overwrite table tab_n9 partition (ds='2008-04-08')
> select key,value from srcbucket_mapjoin_n16;
> select * from
> (select * from tab_n9 where tab_n9.key = 0)a
> join
> (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join
> tab_part_n10 c on a.key = b.key and b.key = c.key
> order by 1,2,3,4,5,6,7,8,9;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[ https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-7: -- > Tez bucket pruning produces wrong result with shared work optimization > -- > > Key: HIVE-7 > URL: https://issues.apache.org/jira/browse/HIVE-7 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22219: --- Attachment: HIVE-22219.patch > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch, HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[
https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934734#comment-16934734
]
Hive QA commented on HIVE-22219:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980840/HIVE-22219.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16801 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=233)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18666/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18666/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18666/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980840 - PreCommit-HIVE-Build
> Bringing a node manager down blocks restart of LLAP service
> ---
>
> Key: HIVE-22219
> URL: https://issues.apache.org/jira/browse/HIVE-22219
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22219.patch
>
>
> For YARN service, when number of running instances != number of desired
> instances, the service state may be in STARTED or FLEX (instead of STABLE).
> For Hive LLAP side, there is a config to control the threshold of service
> health check. The Hive LLAP code misses checking these states, which can
> result in the service not coming up even if the threshold is met.
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Attachment: HIVE-22145.3.patch > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Status: Open (was: Patch Available) > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Status: Patch Available (was: Open) > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Open (was: Patch Available) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Attachment: HIVE-20113.10.patch > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Patch Available (was: Open) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934695#comment-16934695 ] Naveen Gangam commented on HIVE-22217: -- Patch looks good to me. So +1. Thanks [~belugabehr] > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[
https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934691#comment-16934691
]
Hive QA commented on HIVE-22219:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18666/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18666/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bringing a node manager down blocks restart of LLAP service
> ---
>
> Key: HIVE-22219
> URL: https://issues.apache.org/jira/browse/HIVE-22219
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22219.patch
>
>
> For YARN service, when number of running instances != number of desired
> instances, the service state may be in STARTED or FLEX (instead of STABLE).
> For Hive LLAP side, there is a config to control the threshold of service
> health check. The Hive LLAP code misses checking these states, which can
> result in the service not coming up even if the threshold is met.
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934673#comment-16934673
]
Hive QA commented on HIVE-22197:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980839/HIVE-22197.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=361)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=223)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18665/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18665/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18665/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980839 - PreCommit-HIVE-Build
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Patch Available (was: Open)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch,
> HIVE-22079.6.patch, HIVE-22079.7.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Open (was: Patch Available)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch,
> HIVE-22079.6.patch, HIVE-22079.7.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Attachment: HIVE-22079.7.patch
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch,
> HIVE-22079.6.patch, HIVE-22079.7.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: HIVE-5.1.patch Status: Patch Available (was: Open) > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934627#comment-16934627
]
Hive QA commented on HIVE-22197:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 29 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18665/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18665/yetus/whitespace-tabs.txt
|
| modules | C: shims/0.23 ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18665/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20456) Query fails with FNFException using MR with skewjoin enabled and auto convert join disabled
[
https://issues.apache.org/jira/browse/HIVE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934599#comment-16934599
]
Hive QA commented on HIVE-20456:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936968/HIVE-20456.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18664/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18664/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936968 - PreCommit-HIVE-Build
> Query fails with FNFException using MR with skewjoin enabled and auto convert
> join disabled
> ---
>
> Key: HIVE-20456
> URL: https://issues.apache.org/jira/browse/HIVE-20456
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.0, 2.1.1, 3.1.0
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-20456.patch
>
>
> When skew join is enabled and auto convert join is disabled the query fails
> with file not found exception. The following query reproduces the error:
>
> {code:java}
> set hive.optimize.skewjoin = true;
> set hive.auto.convert.join = false;
> set hive.groupby.orderby.position.alias = true;
> set hive.on.master=true;
> set hive.execution.engine=mr;
> set hive.exec.parallel=false;
> drop database if exists test cascade;
> create database if not exists test;
> use test;
> CREATE EXTERNAL TABLE test_table1
> ( `a` int , `b` int, `c` int)
> PARTITIONED BY (
> `d` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> ;
> CREATE EXTERNAL TABLE test_table2
> ( `a` int , `b` int, `c` int)
> PARTITIONED BY (
> `d` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> CREATE EXTERNAL TABLE test_table3
> ( `a` int , `b` int, `c` int)
> PARTITIONED BY (
> `e` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'field.delim'='\u0001',
> 'serialization.format'='\u0001')
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> CREATE EXTERNAL TABLE test_table4 (`a` int , `b` int, `c` int)
> PARTITIONED BY (
> `e` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'field.delim'='\u0001',
> 'serialization.format'='\u0001')
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> with
> temp1 as (
> select
> g.a,
> n.b,
> u.c
> from
> test_table2 g
> inner join test_table4 u on g.a = u.a
> inner join test_table3 n on u.b = n.b
> ),
> temp2 as (
> select
> n.a,
> n.b,
> u.c
> from
> test_table4 u
> inner join test_table3 n on u.b = n.b
> ),
> temp21 as (
> select
> g.b,
> n.c,
> u.a
> from
> temp2 g
> inner join test_table3 u on g.b = u.b
> inner join test_table2 n on u.c = n.c
> group by g.b, n.c, u.a
> ),
> stack as (
> select * from temp1
> union all
> select * from temp21
> )
> select * from stack;
> inner join test_table2 n on u.c = n.c
> group by g.b, n.c, u.a
> ),
> stack as (
> select * from temp1
> union all
> select * from temp21
> )
> select * from stack;
> {code}
> The query runs perfectly fine when tez is used or other combinations of skew
> join and auto convert join are set. On diagnosing the issue, the problem was
> when a conditional task resolves tasks it puts the resolved task directly in
> the runnable state without checking the parental dependencies as well as
> whether the task is already queued.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934520#comment-16934520
]
Dongjoon Hyun commented on HIVE-21237:
--
My bad, [~dawood.m] and guys. I removed my previous comment. Sorry again.
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated HIVE-21237:
-
Comment: was deleted
(was: [~dawood.m]. The following is `Hive` command. If that fails, there is
nothing for Apache Spark community to help you. Please report to Apache Hive
community.
{code}
[root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose
{code})
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934519#comment-16934519
]
Dongjoon Hyun commented on HIVE-21237:
--
[~dawood.m]. The following is `Hive` command. If that fails, there is nothing
for Apache Spark community to help you. Please report to Apache Hive community.
{code}
[root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose
{code}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-5: - > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20456) Query fails with FNFException using MR with skewjoin enabled and auto convert join disabled
[
https://issues.apache.org/jira/browse/HIVE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934548#comment-16934548
]
Hive QA commented on HIVE-20456:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18664/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18664/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query fails with FNFException using MR with skewjoin enabled and auto convert
> join disabled
> ---
>
> Key: HIVE-20456
> URL: https://issues.apache.org/jira/browse/HIVE-20456
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.0, 2.1.1, 3.1.0
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-20456.patch
>
>
> When skew join is enabled and auto convert join is disabled the query fails
> with file not found exception. The following query reproduces the error:
>
> {code:java}
> set hive.optimize.skewjoin = true;
> set hive.auto.convert.join = false;
> set hive.groupby.orderby.position.alias = true;
> set hive.on.master=true;
> set hive.execution.engine=mr;
> set hive.exec.parallel=false;
> drop database if exists test cascade;
> create database if not exists test;
> use test;
> CREATE EXTERNAL TABLE test_table1
> ( `a` int , `b` int, `c` int)
> PARTITIONED BY (
> `d` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> ;
> CREATE EXTERNAL TABLE test_table2
> ( `a` int , `b` int, `c` int)
> PARTITIONED BY (
> `d` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> CREATE EXTERNAL TABLE test_table3
> ( `a` int
[jira] [Updated] (HIVE-22224) Support Parquet-Avro Timestamp Type
[
https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cdmikechen updated HIVE-4:
--
Labels: parquet (was: )
> Support Parquet-Avro Timestamp Type
> ---
>
> Key: HIVE-4
> URL: https://issues.apache.org/jira/browse/HIVE-4
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 2.3.5, 2.3.6
>Reporter: cdmikechen
>Priority: Major
> Labels: parquet
> Fix For: 2.3.7
>
>
> When user create an external table and import a parquet-avro data with 1.8.2
> version which supported logical_type in Hive2.3 or before version, Hive can
> not read timestamp type column data correctly.
> Hive will read it as LongWritable which it actually stores as
> long(logical_type=timestamp-millis).So we may add some codes in
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java
> to let Hive cast long type to timestamp type.
> Some code like below:
>
> public Timestamp getPrimitiveJavaObject(Object o) {
> if (o instanceof LongWritable) {
> return new Timestamp(((LongWritable) o).get());
> }
> return o == null ? null : ((TimestampWritable) o).getTimestamp();
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22224) Support Parquet-Avro Timestamp Type
[
https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cdmikechen reassigned HIVE-4:
-
Assignee: cdmikechen
> Support Parquet-Avro Timestamp Type
> ---
>
> Key: HIVE-4
> URL: https://issues.apache.org/jira/browse/HIVE-4
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 2.3.5, 2.3.6
>Reporter: cdmikechen
>Assignee: cdmikechen
>Priority: Major
> Labels: parquet
> Fix For: 2.3.7
>
>
> When user create an external table and import a parquet-avro data with 1.8.2
> version which supported logical_type in Hive2.3 or before version, Hive can
> not read timestamp type column data correctly.
> Hive will read it as LongWritable which it actually stores as
> long(logical_type=timestamp-millis).So we may add some codes in
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java
> to let Hive cast long type to timestamp type.
> Some code like below:
>
> public Timestamp getPrimitiveJavaObject(Object o) {
> if (o instanceof LongWritable) {
> return new Timestamp(((LongWritable) o).get());
> }
> return o == null ? null : ((TimestampWritable) o).getTimestamp();
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22224) Support Parquet-Avro Timestamp Type
[
https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cdmikechen updated HIVE-4:
--
Description:
When user create an external table and import a parquet-avro data with 1.8.2
version which supported logical_type in Hive2.3 or before version, Hive can not
read timestamp type column data correctly.
Hive will read it as LongWritable which it actually stores as
long(logical_type=timestamp-millis).So we may add some codes in
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java
to let Hive cast long type to timestamp type.
Some code like below:
public Timestamp getPrimitiveJavaObject(Object o) {
if (o instanceof LongWritable) {
return new Timestamp(((LongWritable) o).get());
}
return o == null ? null : ((TimestampWritable) o).getTimestamp();
}
was:
When user create an external table and import a parquet-avro data with 1.8.2
version which supported logical_type in Hive2.3 or before version, Hive can not
read timestamp type column data correctly.
Hive will read it as LongWritable which it actually stores as
long(logical_type=timestamp-millis).So we may add some codes in
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java
to let Hive cast long type to timestamp type.
Some code like below:
public Timestamp getPrimitiveJavaObject(Object o) {
if (o instanceof LongWritable) {
return new Timestamp(((LongWritable) o).get());
}
return o == null ? null : ((TimestampWritable) o).getTimestamp();
}
> Support Parquet-Avro Timestamp Type
> ---
>
> Key: HIVE-4
> URL: https://issues.apache.org/jira/browse/HIVE-4
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 2.3.5, 2.3.6
>Reporter: cdmikechen
>Priority: Major
> Labels: parquet
> Fix For: 2.3.7
>
>
> When user create an external table and import a parquet-avro data with 1.8.2
> version which supported logical_type in Hive2.3 or before version, Hive can
> not read timestamp type column data correctly.
> Hive will read it as LongWritable which it actually stores as
> long(logical_type=timestamp-millis).So we may add some codes in
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java
> to let Hive cast long type to timestamp type.
> Some code like below:
>
> public Timestamp getPrimitiveJavaObject(Object o) {
> if (o instanceof LongWritable) {
> return new Timestamp(((LongWritable) o).get());
> }
> return o == null ? null : ((TimestampWritable) o).getTimestamp();
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Attachment: HIVE-2.01.patch > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Status: Patch Available (was: Open) > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Summary: Clean up the error handling in Driver - get rid of global variables (was: Clean up the error handling in Driver.) > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22222) Clean up the error handling in Driver.
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-2: - > Clean up the error handling in Driver. > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-21975:
Attachment: HIVE-21975.1.patch
> Fix incremental compilation
> ---
>
> Key: HIVE-21975
> URL: https://issues.apache.org/jira/browse/HIVE-21975
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Steve Carlin
>Priority: Major
> Attachments: HIVE-21975.1.patch, HIVE-21975.patch
>
>
> we have an incremental compilation issue around SA ; mostly because of "?
> extends Serializable"
> it could be reproduced with:
> {code}
> git clean -dfx
> mvn install -pl ql -am -DskipTests
> touch `find . -name Sema*A*java` `find . -name Task*Factory.java`
> mvn install -pl ql -DskipTests
> {code}
> error is:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60]
> incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to
> java.util.List>
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49]
> incompatible types: java.util.List>
> cannot be converted to java.util.List extends java.io.Serializable>>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934474#comment-16934474 ] slim bouguerra commented on HIVE-22219: --- +1 > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934475#comment-16934475
]
Dawood commented on HIVE-21237:
---
[~yumwang]on my other setup it fails with below error,
{noformat}
[root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:derby://apachehive:1527/metastore_db;create=true
Metastore Connection Driver :org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
version.
Underlying cause: java.sql.SQLNonTransientConnectionException :
java.net.ConnectException : Error connecting to server apachehive on port 1,527
with message Connection refused (Connection refused).
SQL Error code: 4
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
version.
at
org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:77)
at
org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:144)
at
org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:473)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:577)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:563)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1145)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.sql.SQLNonTransientConnectionException:
java.net.ConnectException : Error connecting to server apachehive on port 1,527
with message Connection refused (Connection refused).
at
org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.client.am.SqlException.getSQLException(Unknown
Source)
at org.apache.derby.jdbc.ClientDriver.connect(Unknown Source)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at
org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:73)
... 11 more
Caused by: ERROR 08001: java.net.ConnectException : Error connecting to server
apachehive on port 1,527 with message Connection refused (Connection refused).
at org.apache.derby.client.net.NetAgent.(Unknown Source)
at org.apache.derby.client.net.NetConnection.newAgent_(Unknown Source)
at org.apache.derby.client.am.ClientConnection.(Unknown Source)
at org.apache.derby.client.net.NetConnection.(Unknown Source)
at
org.apache.derby.client.net.ClientJDBCObjectFactoryImpl.newNetConnection(Unknown
Source)
... 15 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.(Socket.java:436)
at java.base/java.net.Socket.(Socket.java:213)
at
java.base/javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:271)
at org.apache.derby.client.net.OpenSocketAction.run(Unknown Source)
at org.apache.derby.client.net.OpenSocketAction.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
... 20 more
*** schemaTool failed ***
{noformat}
> [JDK 11] SessionState can't be initialized due to
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934466#comment-16934466
]
Ana Jalba commented on HIVE-21508:
--
[~pvary]: This looks like it's similar to the patch for branch-3.1. Would the
same check as before need to be done for ptests to see if the changes are
unrelated to the failing tests?
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.6.branch-3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-1: - Attachment: HIVE-1.1.patch Status: Patch Available (was: Open) > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-1.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?focusedWorklogId=315706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315706 ] ASF GitHub Bot logged work on HIVE-1: - Author: ASF GitHub Bot Created on: 20/Sep/19 14:04 Start Date: 20/Sep/19 14:04 Worklog Time Spent: 10m Work Description: ShubhamChaurasia commented on pull request #778: HIVE-1: Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint URL: https://github.com/apache/hive/pull/778 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315706) Remaining Estimate: 0h Time Spent: 10m > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-1: -- Labels: pull-request-available (was: ) > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia reassigned HIVE-1: > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-21164: Assignee: Laszlo Bodor (was: Vaibhav Gumashta) > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.2.patch, HIVE-21164.3.patch, HIVE-21164.4.patch, > HIVE-21164.5.patch, HIVE-21164.6.patch, HIVE-21164.7.patch, > HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22191) Simplify SemanticAnalyzer by removing unused code
[
https://issues.apache.org/jira/browse/HIVE-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akos Dombi updated HIVE-22191:
--
Status: In Progress (was: Patch Available)
> Simplify SemanticAnalyzer by removing unused code
> -
>
> Key: HIVE-22191
> URL: https://issues.apache.org/jira/browse/HIVE-22191
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Akos Dombi
>Assignee: Akos Dombi
>Priority: Major
> Attachments: HIVE-22191.2.patch, HIVE-22191.patch
>
>
> Simplify {{SemanticAnalyzer}} by:
> - Remove dead code
> - Simplify returning statements
> - Use interfaces types for parameters/fields/variables where it is
> straightforward to migrate
> - Make visibility stricter where it is possible
> - Check logging to use parametrised logging
> - Removing unnecessary keywords (e.g.: {{static}})
> - Some code part could be simplified by using Java 8 features
> I think this is crucial step as this class already contains 15000+ lines of
> code which is screaming for splitting into more reasonable classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-22209:
Attachment: HIVE-22209.1.patch
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.1.patch, HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934329#comment-16934329 ] Ivan Suller commented on HIVE-21000: Created the Accumulo upgrade ticket: HIVE-0 > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22211) Change maven phase to generate test sources
[ https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22211: Attachment: HIVE-22211.01.patch > Change maven phase to generate test sources > --- > > Key: HIVE-22211 > URL: https://issues.apache.org/jira/browse/HIVE-22211 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch, > HIVE-22211.01.patch > > > Some protobuf files are generated in the wrong phase; so I get compile errors > because they are not there for eclipse... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934296#comment-16934296
]
Dawood commented on HIVE-21237:
---
Failed with below error again,
{noformat}
[root@hive apache-hive-2.3.6-bin]# bin/schematool -dbType derby -initSchema
--verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-2.3.6-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:jdbc:derby://hive:1527/metastore_db;create=true
Metastore Connection Driver :org.apache.derby.jdbc.ClientDriver
Metastore connection User: APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.derby.sql
Connecting to jdbc:derby://hive:1527/metastore_db;create=true
Connected to: Apache Derby (version 10.15.1.3 - (1853019))
Driver: Apache Derby Network Client JDBC Driver (version 10.11.1.1 - (1616546))
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:derby://hive:1527/metastore_db> !autocommit on
Autocommit status: true
0: jdbc:derby://hive:1527/metastore_db> CREATE FUNCTION "APP"."NUCLEUS_ASCII"
(C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA
CALLED ON NULL INPUT EXTERNAL NAME
'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii'
Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=3)
Closing: 0: jdbc:derby://hive:1527/metastore_db;create=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:590)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:563)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1145)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:980)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:959)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586)
... 8 more
*** schemaTool failed ***
{noformat}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934284#comment-16934284
]
Hive QA commented on HIVE-22079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980800/HIVE-22079.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=171)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980800 - PreCommit-HIVE-Build
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934274#comment-16934274
]
Yuming Wang edited comment on HIVE-21237 at 9/20/19 10:17 AM:
--
Please try to initialize metastore:
{code:sh}
bin/schematool -dbType derby -initSchema --verbose
{code}
was (Author: q79969786):
Please try to initialize metartore:
{code:sh}
bin/schematool -dbType derby -initSchema --verbose
{code}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934274#comment-16934274
]
Yuming Wang commented on HIVE-21237:
Please try to initialize metartore:
{code:sh}
bin/schematool -dbType derby -initSchema --verbose
{code}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20508) Hive does not support user names of type "user@realm"
[ https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20508: -- Fix Version/s: 4.0.0 > Hive does not support user names of type "user@realm" > - > > Key: HIVE-20508 > URL: https://issues.apache.org/jira/browse/HIVE-20508 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch, > HIVE-20508.3.patch, HIVE-20508.4.patch, HIVE-20508.5.patch, > HIVE-20508.6.patch, HIVE-20508.7.patch > > > Hive does not support user names of type "user@realm". This causes > authorization problems with Ranger for user names containing realms in > Kerberos environment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934252#comment-16934252
]
Hive QA commented on HIVE-22079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 4 new + 104 unchanged - 1
fixed = 108 total (was 105) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
7s{color} | {color:red} ql generated 1 new + 1571 unchanged - 0 fixed = 1572
total (was 1571) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.lib.ExpressionWalker$NodeLabeled be a
_static_ inner class? At ExpressionWalker.java:inner class? At
ExpressionWalker.java:[lines 47-61] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18662/dev-support/hive-personality.sh
|
| git revision | master / 1278bf1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934250#comment-16934250
]
Dawood commented on HIVE-21237:
---
>From the Hive logs,
{noformat}
2019-09-20T05:43:18,483 INFO [main] conf.HiveConf: Found configuration file
file:/opt/apache-hive-2.3.6-bin/conf/hive-site.xml
2019-09-20T05:43:19,231 INFO [main] SessionState:
Logging initialized using configuration in
jar:file:/opt/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar!/hive-log4j2.properties
Async: true
2019-09-20T05:43:19,412 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2019-09-20T05:43:20,466 INFO [main] session.SessionState: Created HDFS
directory: /tmp/hive/root/3515e631-50a7-429e-9875-e1bcc777bb2e
2019-09-20T05:43:20,485 INFO [main] session.SessionState: Created local
directory: /tmp/root/3515e631-50a7-429e-9875-e1bcc777bb2e
2019-09-20T05:43:20,494 INFO [main] session.SessionState: Created HDFS
directory: /tmp/hive/root/3515e631-50a7-429e-9875-e1bcc777bb2e/_tmp_space.db
2019-09-20T05:43:20,512 INFO [main] conf.HiveConf: Using the default value
passed in for log id: 3515e631-50a7-429e-9875-e1bcc777bb2e
2019-09-20T05:43:20,514 INFO [main] session.SessionState: Updating thread name
to 3515e631-50a7-429e-9875-e1bcc777bb2e main
2019-09-20T05:43:20,516 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
2019-09-20T05:43:44,667 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
conf.HiveConf: Using the default value passed in for log id:
3515e631-50a7-429e-9875-e1bcc777bb2e
2019-09-20T05:43:44,732 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
ql.Driver: Compiling
command(queryId=root_20190920054344_d4447492-7063-4d08-a7f3-9ba47b1dc636): show
tables
2019-09-20T05:43:45,175 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
metastore.HiveMetaStore: 0: Opening raw store with implementation
class:org.apache.hadoop.hive.metastore.ObjectStore
2019-09-20T05:43:45,210 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
metastore.ObjectStore: ObjectStore, initialize called
2019-09-20T05:43:46,221 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main]
metastore.ObjectStore: Setting MetaStore object pin classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2019-09-20T05:43:48,045 WARN [3515e631-50a7-429e-9875-e1bcc777bb2e main]
metastore.MetaStoreDirectSql: Self-test query [select "DB_ID" from "DBS"]
failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from
"DBS"".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.runTestQuery(MetaStoreDirectSql.java:230)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:144)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:410)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:342)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:303)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
~[hadoop-common-3.2.0.jar:?]
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
~[hadoop-common-3.2.0.jar:?]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:58)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:628)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:594)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:588)
~[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:655)
~[hive-exec-2.3.6.jar:2.3.6]
at
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934213#comment-16934213
]
Yuming Wang commented on HIVE-21237:
Please check the Hive log: \{{/tmp/${USER}/hive.log}}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934203#comment-16934203
]
Dawood commented on HIVE-21237:
---
Hello [~yumwang]
Now I get the below error on my hive setup,
{noformat}
hive> show tables;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
{noformat}
Could you please help here, thx
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Component/s: Standalone Metastore Metastore > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive, Metastore, Standalone Metastore >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Fix Version/s: 4.0.0 > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934200#comment-16934200 ] Laszlo Bodor commented on HIVE-22202: - pushed to master, thanks [~ghanko] for the patch! > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?focusedWorklogId=315518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315518 ] ASF GitHub Bot logged work on HIVE-22218: - Author: ASF GitHub Bot Created on: 20/Sep/19 08:06 Start Date: 20/Sep/19 08:06 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #777: HIVE-22218 Break up DDLSemanticAnalyzer - extract Workload Management… URL: https://github.com/apache/hive/pull/777 … related analyzers This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315518) Remaining Estimate: 0h Time Spent: 10m > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22218.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #5: extract all the workload management related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934166#comment-16934166 ] Ivan Suller commented on HIVE-21000: I don't know if I need to worry or not, but I do. It doesn't sound promising, but we will see how it'll turn out. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934161#comment-16934161
]
David Lavati commented on HIVE-21000:
-
{quote}Another thing that might need investigation, is the fact that the fb303
wrapper according to THRIFT-4506 is no longer maintained in the Thrift ASF
project. A high number of handlers and metastore-related scripts rely on this.
{quote}
Is this something we need to worry about? I'm not sure how it will play along
with a newer thrift version.
> Upgrade thrift to at least 0.10.0
> -
>
> Key: HIVE-21000
> URL: https://issues.apache.org/jira/browse/HIVE-21000
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Ivan Suller
>Priority: Major
> Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch,
> HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch,
> HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch,
> sampler_before.png
>
>
> I was looking into some compile profiles for tables with lots of columns; and
> it turned out that [thrift 0.9.3 is allocating a
> List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348]
> during every hashcode calculation; but luckily THRIFT-2877 is improving on
> that - so I propose to upgrade to at least 0.10.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934160#comment-16934160 ] Ivan Suller commented on HIVE-21000: Not yet, but will open a ticket for Accumulo upgrade. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[
https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934158#comment-16934158
]
Hive QA commented on HIVE-20113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980777/HIVE-20113.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16803 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=177)
[groupby_rollup_empty.q,schema_evol_orc_acidvec_table_llap_io.q,parquet_types.q,vectorization_5.q,schema_evol_stats.q,subquery_corr.q,jdbc_handler.q,column_access_stats.q,vector_between_in.q,merge1.q,cbo_rp_join.q,subquery_ANY.q,vector_coalesce_3.q,sample10_mm.q,stats_noscan_1.q,vector_partition_diff_num_cols.q,vector_decimal_10_0.q,orc_merge5.q,schema_evol_text_vec_part.q,vector_groupby_reduce.q,union8.q,lineage2.q,auto_join29.q,insert_values_orig_table.q,correlationoptimizer1.q,vector_decimal_mapjoin.q,schema_evol_undecorated.q,vectorized_case.q,vector_windowing_rank.q,materialized_view_rewrite_8.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18661/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18661/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980777 - PreCommit-HIVE-Build
> Shuffle avoidance: Disable 1-1 edges for sorted shuffle
>
>
> Key: HIVE-20113
> URL: https://issues.apache.org/jira/browse/HIVE-20113
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch,
> HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch,
> HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch,
> HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch
>
>
> The sorted shuffle avoidance can have some issues when the shuffle data gets
> broken up into multiple chunks on disk.
> The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to
> have a final merge at all, it should open a single compressed file and write
> a single index entry.
> Until the shuffle issue is resolved & a lot more testing, it is prudent to
> disable the optimization for sorted shuffle edges and stop rewriting the
> RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934150#comment-16934150 ] Zoltan Haindrich edited comment on HIVE-21000 at 9/20/19 7:19 AM: -- awesome :D do you have a separate ticket for the accumulo-2 upgrade? if not I think we would need one - just to make it more clear what is happening :) was (Author: kgyrtkirk): awesome :D do you have a separate ticket for the accumulo-2 upgrade? > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934150#comment-16934150 ] Zoltan Haindrich commented on HIVE-21000: - awesome :D do you have a separate ticket for the accumulo-2 upgrade? > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Status: Patch Available (was: Open) The failed tests are passing locally with hbase-2.0.0-alpha4. The prior patch was using 2.1.0 of hbase. Perhaps related to these failures. > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Attachment: HIVE-22205.4.patch > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Status: Patch Available (was: In Progress) > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LuGuangMing updated HIVE-22098:
---
Status: Patch Available (was: Reopened)
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Major
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and reducers number greater
> than 2, result is easy to lose data.
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: HIVE-22198.patch > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934132#comment-16934132 ] LuGuangMing commented on HIVE-22198: 【repair】 Fix the parent dependencies of children stage-3 when stage-2 and stage-6 run，details check [^HIVE-22198.patch] > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Affects Version/s: (was: 3.0.0) (was: 1.2.0) > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Description: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment [^test-parallel.sql] create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; The result data should be two, but only one. was: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; The result data should be two, but only one. > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; >
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Description: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; The result data should be two, but only one. was: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment sql file create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; The result data should be two, but only one. > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two,
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[
https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934116#comment-16934116
]
Hive QA commented on HIVE-20113:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 347 unchanged - 1
fixed = 348 total (was 348) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18661/dev-support/hive-personality.sh
|
| git revision | master / 0c1ef6f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18661/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18661/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shuffle avoidance: Disable 1-1 edges for sorted shuffle
>
>
> Key: HIVE-20113
> URL: https://issues.apache.org/jira/browse/HIVE-20113
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Labels: Branch3Candidate
> Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch,
> HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch,
> HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch,
> HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch
>
>
> The sorted shuffle avoidance can have some issues when the shuffle data gets
> broken up into multiple chunks on disk.
> The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to
> have a final merge at all, it should open a single compressed file and write
> a single index entry.
> Until the shuffle issue is resolved & a lot more testing, it is prudent to
> disable the optimization for sorted shuffle edges and stop rewriting the
> RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
