[jira] [Commented] (HIVE-23536) Provide an option to skip stats generation for major compaction
[ https://issues.apache.org/jira/browse/HIVE-23536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115768#comment-17115768 ] Ashutosh Chauhan commented on HIVE-23536: - +1 > Provide an option to skip stats generation for major compaction > --- > > Key: HIVE-23536 > URL: https://issues.apache.org/jira/browse/HIVE-23536 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23536.02.patch, HIVE-23536.03.patch, > HIVE-23536.patch > > > Currently major MR compaction is regenerates stats every time if the column > stats table contains some data. Some configurations do not use stats but > because of historical reasons the column stats table can still contain some > data. > We should provide a possibility to skip stats generation in these cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23480) Test may fail due to a incorrect usage of a third party library
[
https://issues.apache.org/jira/browse/HIVE-23480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23480:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Panos!
> Test may fail due to a incorrect usage of a third party library
> ---
>
> Key: HIVE-23480
> URL: https://issues.apache.org/jira/browse/HIVE-23480
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore
>Reporter: contextshuffling
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23480.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Tests
> {{org.apache.hadoop.hive.common.TestStatsSetupConst#testStatColumnEntriesCompat}}
> replies on Jackson to serialize the params to string. However, Jackson
> library uses reflection API {{getDeclaredFields}} but it does not guarantee
> any specific order of returned field so the order of fields in the json
> string might change, and thus, test can fail or pass without any changes to
> the code.
> An example error message:
> {code:java}
> org.junit.ComparisonFailure:
> expected:<{"[BASIC_STATS":"true","COLUMN_STATS":{"Foo":"true"}]}> but
> was:<{"[COLUMN_STATS":{"Foo":"true"},"BASIC_STATS":"true"]}>
> at
> org.apache.hadoop.hive.common.TestStatsSetupConst.testStatColumnEntriesCompat(TestStatsSetupConst.java:76)
> {code}
> Ideally, this test should not reply on the order returned by this API so that
> it generates a deterministic result.
> An potential solution is to use library like
> https://github.com/skyscreamer/JSONassert to compare string in a
> order-agnostic way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23526) Out of sequence seen in Beeline may swallow the real problem
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115748#comment-17115748 ] Zhihua Deng commented on HIVE-23526: [~thejas], [~pvary], [~ashutoshc] could you take a look? thanks. > Out of sequence seen in Beeline may swallow the real problem > - > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with verbose > flag, this makes problem puzzled as beeline does not have concurrency problem > on underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-23526) Out of sequence seen in Beeline may swallow the real problem
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Comment: was deleted (was: [~thejas] [~pvary] [~ashutoshc] any comments here? ) > Out of sequence seen in Beeline may swallow the real problem > - > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with verbose > flag, this makes problem puzzled as beeline does not have concurrency problem > on underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23517) Update tpcds queries: q4 q11 q74
[ https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23517: -- Status: Patch Available (was: Open) > Update tpcds queries: q4 q11 q74 > > > Key: HIVE-23517 > URL: https://issues.apache.org/jira/browse/HIVE-23517 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch, > HIVE-23517.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23517) Update tpcds queries: q4 q11 q74
[ https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23517: -- Attachment: HIVE-23517.1.patch > Update tpcds queries: q4 q11 q74 > > > Key: HIVE-23517 > URL: https://issues.apache.org/jira/browse/HIVE-23517 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch, > HIVE-23517.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23517) Update tpcds queries: q4 q11 q74
[
https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115762#comment-17115762
]
Hive QA commented on HIVE-23517:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22600/dev-support/hive-personality.sh
|
| git revision | master / 3015ff2 |
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22600/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update tpcds queries: q4 q11 q74
>
>
> Key: HIVE-23517
> URL: https://issues.apache.org/jira/browse/HIVE-23517
> Project: Hive
> Issue Type: Task
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115759#comment-17115759
]
Hive QA commented on HIVE-23491:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003900/HIVE-23491.3.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17283 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22599/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22599/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22599/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003900 - PreCommit-HIVE-Build
> Move ParseDriver to parser module
> -
>
> Key: HIVE-23491
> URL: https://issues.apache.org/jira/browse/HIVE-23491
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch,
> HIVE-23491.3.patch
>
>
> Move *ParseDriver* class and syntax Parsing related unit tests to the parse
> module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23517) Update tpcds queries: q4 q11 q74
[ https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23517: -- Status: Open (was: Patch Available) > Update tpcds queries: q4 q11 q74 > > > Key: HIVE-23517 > URL: https://issues.apache.org/jira/browse/HIVE-23517 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch, > HIVE-23517.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23480) Test may fail due to a incorrect usage of a third party library
[
https://issues.apache.org/jira/browse/HIVE-23480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115774#comment-17115774
]
Ashutosh Chauhan commented on HIVE-23480:
-
+1
> Test may fail due to a incorrect usage of a third party library
> ---
>
> Key: HIVE-23480
> URL: https://issues.apache.org/jira/browse/HIVE-23480
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore
>Reporter: contextshuffling
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Attachments: HIVE-23480.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Tests
> {{org.apache.hadoop.hive.common.TestStatsSetupConst#testStatColumnEntriesCompat}}
> replies on Jackson to serialize the params to string. However, Jackson
> library uses reflection API {{getDeclaredFields}} but it does not guarantee
> any specific order of returned field so the order of fields in the json
> string might change, and thus, test can fail or pass without any changes to
> the code.
> An example error message:
> {code:java}
> org.junit.ComparisonFailure:
> expected:<{"[BASIC_STATS":"true","COLUMN_STATS":{"Foo":"true"}]}> but
> was:<{"[COLUMN_STATS":{"Foo":"true"},"BASIC_STATS":"true"]}>
> at
> org.apache.hadoop.hive.common.TestStatsSetupConst.testStatColumnEntriesCompat(TestStatsSetupConst.java:76)
> {code}
> Ideally, this test should not reply on the order returned by this API so that
> it generates a deterministic result.
> An potential solution is to use library like
> https://github.com/skyscreamer/JSONassert to compare string in a
> order-agnostic way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115741#comment-17115741
]
Hive QA commented on HIVE-23491:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
56s{color} | {color:blue} itests/util in master has 54 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} parser: The patch generated 194 new + 0 unchanged - 0
fixed = 194 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
3s{color} | {color:red} ql: The patch generated 1 new + 650 unchanged - 195
fixed = 651 total (was 845) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 1 new + 18 unchanged
- 0 fixed = 19 total (was 18) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22599/dev-support/hive-personality.sh
|
| git revision | master / 3015ff2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22599/yetus/diff-checkstyle-parser.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22599/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22599/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22599/yetus/whitespace-eol.txt
|
| modules | C: parser ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22599/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically
[jira] [Updated] (HIVE-23491) Move ParseDriver to parser module
[ https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23491: -- Status: Open (was: Patch Available) > Move ParseDriver to parser module > - > > Key: HIVE-23491 > URL: https://issues.apache.org/jira/browse/HIVE-23491 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch, > HIVE-23491.3.patch, HIVE-23491.4.patch > > > Move *ParseDriver* class and syntax Parsing related unit tests to the parse > module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23491) Move ParseDriver to parser module
[ https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23491: -- Attachment: HIVE-23491.4.patch > Move ParseDriver to parser module > - > > Key: HIVE-23491 > URL: https://issues.apache.org/jira/browse/HIVE-23491 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch, > HIVE-23491.3.patch, HIVE-23491.4.patch > > > Move *ParseDriver* class and syntax Parsing related unit tests to the parse > module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23491) Move ParseDriver to parser module
[ https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23491: -- Status: Patch Available (was: Open) > Move ParseDriver to parser module > - > > Key: HIVE-23491 > URL: https://issues.apache.org/jira/browse/HIVE-23491 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch, > HIVE-23491.3.patch, HIVE-23491.4.patch > > > Move *ParseDriver* class and syntax Parsing related unit tests to the parse > module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115912#comment-17115912
]
Hive QA commented on HIVE-23468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003904/HIVE-23468.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17283 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=227)
org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.testHookContexts (batchId=237)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22602/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22602/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22602/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003904 - PreCommit-HIVE-Build
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23546: -- Labels: pull-request-available (was: ) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?focusedWorklogId=437136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437136 ] ASF GitHub Bot logged work on HIVE-23546: - Author: ASF GitHub Bot Created on: 25/May/20 11:39 Start Date: 25/May/20 11:39 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1033: URL: https://github.com/apache/hive/pull/1033 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 437136) Remaining Estimate: 0h Time Spent: 10m > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115901#comment-17115901
]
Hive QA commented on HIVE-23468:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22602/dev-support/hive-personality.sh
|
| git revision | master / ef7a9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22602/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: (was: HIVE-23546.patch) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: HIVE-23546.patch > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > Attachments: HIVE-23546.patch > > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: (was: HIVE-23546.patch) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23394) TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky
[
https://issues.apache.org/jira/browse/HIVE-23394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115949#comment-17115949
]
Zoltan Haindrich commented on HIVE-23394:
-
statements in
itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractTestJdbcGenericUDTFGetSplits.java
are not closed properly - however fixing that doesn't fix the issue...
> TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky
>
>
> Key: HIVE-23394
> URL: https://issues.apache.org/jira/browse/HIVE-23394
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Priority: Major
>
> both
> TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1 and
> TestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1
> can fail with the exception below
> seems like the connection was lost
> {code}
> Error Message
> Failed to close statement
> Stacktrace
> java.sql.SQLException: Failed to close statement
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:200)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:205)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.AbstractTestJdbcGenericUDTFGetSplits.runQuery(AbstractTestJdbcGenericUDTFGetSplits.java:135)
> at
> org.apache.hive.jdbc.AbstractTestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1(AbstractTestJdbcGenericUDTFGetSplits.java:164)
> at
> org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1(TestJdbcGenericUDTFGetSplits2.java:28)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: org.apache.thrift.TApplicationException: CloseOperation failed:
> out of sequence response
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1732)
> at com.sun.proxy.$Proxy146.CloseOperation(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:193)
> ... 14 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Comment: was deleted (was: [~pgaref] Could you take some time to look at this? thanks.) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23517) Update tpcds queries: q4 q11 q74
[
https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115804#comment-17115804
]
Hive QA commented on HIVE-23517:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003899/HIVE-23517.1.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17283 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22600/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22600/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22600/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003899 - PreCommit-HIVE-Build
> Update tpcds queries: q4 q11 q74
>
>
> Key: HIVE-23517
> URL: https://issues.apache.org/jira/browse/HIVE-23517
> Project: Hive
> Issue Type: Task
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch,
> HIVE-23517.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: HIVE-23546.patch > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > Attachments: HIVE-23546.patch > > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115955#comment-17115955 ] Zhihua Deng commented on HIVE-23546: [~pgaref] Could you take some time to look at this? thanks. > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23536) Provide an option to skip stats generation for major compaction
[
https://issues.apache.org/jira/browse/HIVE-23536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115936#comment-17115936
]
Hive QA commented on HIVE-23536:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 1 new + 49 unchanged - 2 fixed
= 50 total (was 51) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22603/dev-support/hive-personality.sh
|
| git revision | master / ef7a9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22603/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22603/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide an option to skip stats generation for major compaction
> ---
>
> Key: HIVE-23536
> URL: https://issues.apache.org/jira/browse/HIVE-23536
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23536.02.patch, HIVE-23536.03.patch,
> HIVE-23536.patch
>
>
> Currently major MR compaction is regenerates stats every time if the column
> stats table contains some data. Some configurations do not use stats but
> because of historical reasons the column stats table can still contain some
> data.
> We should provide a possibility to skip stats generation in these cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23524) TestNewGetSplitsFormat is unstable
[
https://issues.apache.org/jira/browse/HIVE-23524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115946#comment-17115946
]
Zoltan Haindrich edited comment on HIVE-23524 at 5/25/20, 11:34 AM:
same goes for:
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNewGetSplitsFormatReturnPath.java
was (Author: kgyrtkirk):
probably the same goes for:
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNewGetSplitsFormatReturnPath.java
> TestNewGetSplitsFormat is unstable
> --
>
> Key: HIVE-23524
> URL: https://issues.apache.org/jira/browse/HIVE-23524
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Priority: Major
>
> http://34.66.156.144:8080/job/hive-c/5/console
> {code}
> 12:05:16 [INFO] ---
> 12:05:16 [INFO] T E S T S
> 12:05:16 [INFO] ---
> 12:05:16 [INFO] Running org.apache.hive.jdbc.TestNewGetSplitsFormat
> 12:12:38 [ERROR] Tests run: 9, Failures: 0, Errors: 2, Skipped: 0, Time
> elapsed: 433.083 s <<< FAILURE! - in
> org.apache.hive.jdbc.TestNewGetSplitsFormat
> 12:12:38 [ERROR]
> testLlapInputFormatEndToEnd(org.apache.hive.jdbc.TestNewGetSplitsFormat)
> Time elapsed: 82.662 s <<< ERROR!
> 12:12:38 java.io.IOException: java.sql.SQLException: Error while cleaning up
> the server resources
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:350)
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapArrowRowInputFormat.getSplits(LlapArrowRowInputFormat.java:54)
> 12:12:38 at
> org.apache.hive.jdbc.TestNewGetSplitsFormat.processQuery(TestNewGetSplitsFormat.java:87)
> 12:12:38 at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.processQuery(BaseJdbcWithMiniLlap.java:676)
> 12:12:38 at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.testLlapInputFormatEndToEnd(BaseJdbcWithMiniLlap.java:223)
> 12:12:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 12:12:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 12:12:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 12:12:38 at java.lang.reflect.Method.invoke(Method.java:498)
> 12:12:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 12:12:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 12:12:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 12:12:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 12:12:38 at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> 12:12:38 Caused by: java.sql.SQLException: Error while cleaning up the
> server resources
> 12:12:38 at
> org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:1018)
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:342)
> 12:12:38 ... 13 more
> 12:12:38 Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> 12:12:38 at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> 12:12:38 at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
> 12:12:38 at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> 12:12:38 at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> 12:12:38 at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
> 12:12:38 at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
> 12:12:38 at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> 12:12:38 at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:199)
> 12:12:38 at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:186)
> 12:12:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 12:12:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 12:12:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 12:12:38
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Status: Patch Available (was: Open) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: HIVE-23546.patch > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > Attachments: HIVE-23546.patch > > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: (was: HIVE-23546.patch) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: HIVE-23546.patch > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Attachments: HIVE-23546.patch > > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[
https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115824#comment-17115824
]
Hive QA commented on HIVE-21624:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} llap-server: The patch generated 1 new + 38 unchanged
- 0 fixed = 39 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22601/dev-support/hive-personality.sh
|
| git revision | master / ef7a9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22601/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22601/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Cpu metrics at thread level is broken
> ---
>
> Key: HIVE-21624
> URL: https://issues.apache.org/jira/browse/HIVE-21624
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu
> metrics when available. At some point, the thread name which the metrics
> publisher looks for has changed causing no metrics to be published for these
> counters.
> The above counters looks for thread with name starting with
> "ContainerExecutor" but the llap task executor thread got changed to
> "Task-Executor"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[
https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115866#comment-17115866
]
Hive QA commented on HIVE-21624:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003901/HIVE-21624.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17283 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=130)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22601/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22601/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003901 - PreCommit-HIVE-Build
> LLAP: Cpu metrics at thread level is broken
> ---
>
> Key: HIVE-21624
> URL: https://issues.apache.org/jira/browse/HIVE-21624
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu
> metrics when available. At some point, the thread name which the metrics
> publisher looks for has changed causing no metrics to be published for these
> counters.
> The above counters looks for thread with name starting with
> "ContainerExecutor" but the llap task executor thread got changed to
> "Task-Executor"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: (was: HIVE-23546.patch) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Attachment: HIVE-23546.patch > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23524) TestNewGetSplitsFormat is unstable
[
https://issues.apache.org/jira/browse/HIVE-23524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115946#comment-17115946
]
Zoltan Haindrich commented on HIVE-23524:
-
probably the same goes for:
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestNewGetSplitsFormatReturnPath.java
> TestNewGetSplitsFormat is unstable
> --
>
> Key: HIVE-23524
> URL: https://issues.apache.org/jira/browse/HIVE-23524
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Priority: Major
>
> http://34.66.156.144:8080/job/hive-c/5/console
> {code}
> 12:05:16 [INFO] ---
> 12:05:16 [INFO] T E S T S
> 12:05:16 [INFO] ---
> 12:05:16 [INFO] Running org.apache.hive.jdbc.TestNewGetSplitsFormat
> 12:12:38 [ERROR] Tests run: 9, Failures: 0, Errors: 2, Skipped: 0, Time
> elapsed: 433.083 s <<< FAILURE! - in
> org.apache.hive.jdbc.TestNewGetSplitsFormat
> 12:12:38 [ERROR]
> testLlapInputFormatEndToEnd(org.apache.hive.jdbc.TestNewGetSplitsFormat)
> Time elapsed: 82.662 s <<< ERROR!
> 12:12:38 java.io.IOException: java.sql.SQLException: Error while cleaning up
> the server resources
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:350)
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapArrowRowInputFormat.getSplits(LlapArrowRowInputFormat.java:54)
> 12:12:38 at
> org.apache.hive.jdbc.TestNewGetSplitsFormat.processQuery(TestNewGetSplitsFormat.java:87)
> 12:12:38 at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.processQuery(BaseJdbcWithMiniLlap.java:676)
> 12:12:38 at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.testLlapInputFormatEndToEnd(BaseJdbcWithMiniLlap.java:223)
> 12:12:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 12:12:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 12:12:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 12:12:38 at java.lang.reflect.Method.invoke(Method.java:498)
> 12:12:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 12:12:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 12:12:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 12:12:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 12:12:38 at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> 12:12:38 Caused by: java.sql.SQLException: Error while cleaning up the
> server resources
> 12:12:38 at
> org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:1018)
> 12:12:38 at
> org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:342)
> 12:12:38 ... 13 more
> 12:12:38 Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> 12:12:38 at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> 12:12:38 at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
> 12:12:38 at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
> 12:12:38 at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> 12:12:38 at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> 12:12:38 at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
> 12:12:38 at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
> 12:12:38 at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> 12:12:38 at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:199)
> 12:12:38 at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:186)
> 12:12:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 12:12:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 12:12:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 12:12:38 at java.lang.reflect.Method.invoke(Method.java:498)
> 12:12:38 at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1732)
> 12:12:38 at
[jira] [Commented] (HIVE-23536) Provide an option to skip stats generation for major compaction
[
https://issues.apache.org/jira/browse/HIVE-23536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115945#comment-17115945
]
Hive QA commented on HIVE-23536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003903/HIVE-23536.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17284 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22603/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22603/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22603/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003903 - PreCommit-HIVE-Build
> Provide an option to skip stats generation for major compaction
> ---
>
> Key: HIVE-23536
> URL: https://issues.apache.org/jira/browse/HIVE-23536
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23536.02.patch, HIVE-23536.03.patch,
> HIVE-23536.patch
>
>
> Currently major MR compaction is regenerates stats every time if the column
> stats table contains some data. Some configurations do not use stats but
> because of historical reasons the column stats table can still contain some
> data.
> We should provide a possibility to skip stats generation in these cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23546: - Assignee: Zhihua Deng > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser use show partitions to determine whether is OK to add partitions > when the external data is ready, or take a work flow one step further in a > busy hive cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23544:
Attachment: HIVE-23544.01.patch
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerLoadTask.java:[147,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_LOAD
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115984#comment-17115984
]
Hive QA commented on HIVE-23491:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
53s{color} | {color:blue} itests/util in master has 54 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} parser: The patch generated 193 new + 0 unchanged - 0
fixed = 193 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} ql: The patch generated 0 new + 649 unchanged - 196
fixed = 649 total (was 845) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22604/dev-support/hive-personality.sh
|
| git revision | master / ef7a9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22604/yetus/diff-checkstyle-parser.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22604/yetus/whitespace-eol.txt
|
| modules | C: parser ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move ParseDriver to parser module
> -
>
> Key: HIVE-23491
> URL: https://issues.apache.org/jira/browse/HIVE-23491
> Project: Hive
> Issue Type:
[jira] [Commented] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116078#comment-17116078
]
Hive QA commented on HIVE-23544:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22606/dev-support/hive-personality.sh
|
| git revision | master / 9ab45e2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22606/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
>
[jira] [Updated] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[ https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23547: -- Attachment: HIVE-23547.01.patch > Enforce testconfiguration.properties file format and alphabetical order > --- > > Key: HIVE-23547 > URL: https://issues.apache.org/jira/browse/HIVE-23547 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > Attachments: HIVE-23547.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[ https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23547: -- Status: Patch Available (was: Open) > Enforce testconfiguration.properties file format and alphabetical order > --- > > Key: HIVE-23547 > URL: https://issues.apache.org/jira/browse/HIVE-23547 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > Attachments: HIVE-23547.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23544:
Attachment: HIVE-23544.01.patch
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch,
> HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerLoadTask.java:[147,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_LOAD
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22942: Status: Patch Available (was: Open) > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22942.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework > some info/etc about how it compares to existing one: > https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23546) Skip authorization when user is a superuser
[
https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116037#comment-17116037
]
Hive QA commented on HIVE-23546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003943/HIVE-23546.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17279 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=186)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22605/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22605/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22605/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003943 - PreCommit-HIVE-Build
> Skip authorization when user is a superuser
> ---
>
> Key: HIVE-23546
> URL: https://issues.apache.org/jira/browse/HIVE-23546
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23546.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If the current user is a superuser, there is no need to do authorization.
> This can speed up queries, especially for those ddl queries. For example, the
> superuser add partitions when the external data is ready, or show partitions
> to check whether it OK to take the work flow one step further in a busy hive
> cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23517) Update perf driver queries: q4 q11 q74
[ https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23517: --- Summary: Update perf driver queries: q4 q11 q74 (was: Update tpcds queries: q4 q11 q74) > Update perf driver queries: q4 q11 q74 > -- > > Key: HIVE-23517 > URL: https://issues.apache.org/jira/browse/HIVE-23517 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch, > HIVE-23517.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23530:
---
Attachment: HIVE-23530.02.patch
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.patch
>
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23546: --- Description: If the current user is a superuser, there is no need to do authorization. This can speed up queries, especially for those ddl queries. For example, the superuser add partitions when the external data is ready, or show partitions to check whether it OK to take the work flow one step further in a busy hive cluster. (was: If the current user is a superuser, there is no need to do authorization. This can speed up queries, especially for those ddl queries. For example, the superuser use show partitions to determine whether is OK to add partitions when the external data is ready, or take a work flow one step further in a busy hive cluster.) > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser add partitions when the external data is ready, or show partitions > to check whether it OK to take the work flow one step further in a busy hive > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22942: Attachment: HIVE-22942.01.patch > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22942.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework > some info/etc about how it compares to existing one: > https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23517) Update perf driver queries: q4 q11 q74
[ https://issues.apache.org/jira/browse/HIVE-23517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23517: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~kkasa]! > Update perf driver queries: q4 q11 q74 > -- > > Key: HIVE-23517 > URL: https://issues.apache.org/jira/browse/HIVE-23517 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-23517.1.patch, HIVE-23517.1.patch, > HIVE-23517.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23546) Skip authorization when user is a superuser
[
https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116022#comment-17116022
]
Hive QA commented on HIVE-23546:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22605/dev-support/hive-personality.sh
|
| git revision | master / 9ab45e2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22605/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip authorization when user is a superuser
> ---
>
> Key: HIVE-23546
> URL: https://issues.apache.org/jira/browse/HIVE-23546
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23546.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If the current user is a superuser, there is no need to do authorization.
> This can speed up queries, especially for those ddl queries. For example, the
> superuser add partitions when the external data is ready, or show partitions
> to check whether it OK to take the work flow one step further in a busy hive
> cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116101#comment-17116101
]
Hive QA commented on HIVE-23544:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003947/HIVE-23544.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17283 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.auth.TestImproperTrustDomainAuthenticationBinary.org.apache.hive.service.auth.TestImproperTrustDomainAuthenticationBinary
(batchId=211)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22606/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22606/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22606/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003947 - PreCommit-HIVE-Build
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerLoadTask.java:[147,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_LOAD
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23491) Move ParseDriver to parser module
[
https://issues.apache.org/jira/browse/HIVE-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116000#comment-17116000
]
Hive QA commented on HIVE-23491:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003915/HIVE-23491.4.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17283 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003915 - PreCommit-HIVE-Build
> Move ParseDriver to parser module
> -
>
> Key: HIVE-23491
> URL: https://issues.apache.org/jira/browse/HIVE-23491
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23491.2.patch, HIVE-23491.3.patch,
> HIVE-23491.3.patch, HIVE-23491.4.patch
>
>
> Move *ParseDriver* class and syntax Parsing related unit tests to the parse
> module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22942) Replace PTest with an alternative
[
https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116133#comment-17116133
]
Hive QA commented on HIVE-22942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003951/HIVE-22942.01.patch
{color:green}SUCCESS:{color} +1 due to 24 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17230 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003951 - PreCommit-HIVE-Build
> Replace PTest with an alternative
> -
>
> Key: HIVE-22942
> URL: https://issues.apache.org/jira/browse/HIVE-22942
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22942.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I never opened a jira about this...but it might actually help collect ideas
> and actually start going somewhere sooner than later :D
> Right now we maintain the ptest2 project inside Hive to be able to run Hive
> tests in a distributed fashion...the backstab of this solution is that we are
> putting much effort into maintaining a distributed test execution framework...
> I think it would be better if we could find an off the shelf solution for the
> task and migrate to that instead of putting more efforts into the ptest
> framework
> some info/etc about how it compares to existing one:
> https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116180#comment-17116180
]
Hive QA commented on HIVE-23530:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} accumulo-handler in master has 20 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} contrib in master has 11 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} ql: The patch generated 18 new + 144 unchanged - 59
fixed = 162 total (was 203) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22610/dev-support/hive-personality.sh
|
| git revision | master / 9075e26 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22610/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql accumulo-handler contrib hbase-handler itests/hive-blobstore
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22610/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.patch
>
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the
[jira] [Commented] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116186#comment-17116186
]
Hive QA commented on HIVE-23530:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003959/HIVE-23530.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=25)
[script_pipe.q,parquet_complex_types_vectorization.q,vector_custom_udf_configure.q,autoColumnStats_6.q,mapreduce1.q,explainuser_2.q,mm_all.q,orc_merge4.q,dynamic_semijoin_user_level.q,orc_struct_type_vectorization.q,union_script.q,external_table_with_space_in_location_path.q,udf_printf.q,external_table_purge.q,insert_into6.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_cluster]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_distribute_sort]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partition_cluster]
(batchId=110)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=118)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned_3]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_window]
(batchId=107)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22610/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22610/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003959 - PreCommit-HIVE-Build
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.patch
>
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116154#comment-17116154
]
Hive QA commented on HIVE-23544:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
29s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22609/dev-support/hive-personality.sh
|
| git revision | master / 9075e26 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22609/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch,
> HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
[jira] [Updated] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[ https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23547: -- Attachment: HIVE-23547.02.patch > Enforce testconfiguration.properties file format and alphabetical order > --- > > Key: HIVE-23547 > URL: https://issues.apache.org/jira/browse/HIVE-23547 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > Attachments: HIVE-23547.01.patch, HIVE-23547.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[
https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116189#comment-17116189
]
Hive QA commented on HIVE-23547:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003967/HIVE-23547.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22611/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22611/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22611/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:00:46.897
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22611/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:00:46.900
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:00:48.216
+ rm -rf ../yetus_PreCommit-HIVE-Build-22611
+ mkdir ../yetus_PreCommit-HIVE-Build-22611
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22611
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22611/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4886477388448019212.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc4886477388448019212.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc2379701296703047080.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveLexer.g
Output file
[jira] [Commented] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[
https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116141#comment-17116141
]
Hive QA commented on HIVE-23547:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003956/HIVE-23547.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-25 16:19:17.979
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22608/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 16:19:17.985
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
9ab45e2..9075e26 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 9ab45e2 HIVE-23510 TestMiniLlapLocalCliDriver should be the
default driver for q tests ADDENDUM remove row left in accidentally (Miklos
Gergely, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 16:19:20.175
+ rm -rf ../yetus_PreCommit-HIVE-Build-22608
+ mkdir ../yetus_PreCommit-HIVE-Build-22608
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22608
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22608/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8271270571614104874.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc8271270571614104874.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc2619960140621441972.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
[jira] [Updated] (HIVE-23365) Put RS deduplication optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-23365: --- Attachment: HIVE-23365.02.patch > Put RS deduplication optimization under cost based decision > --- > > Key: HIVE-23365 > URL: https://issues.apache.org/jira/browse/HIVE-23365 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Stamatis Zampetakis >Priority: Major > Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch > > > Currently, RS deduplication is always executed whenever it is semantically > correct. However, it could be beneficial to leave both RS operators in the > plan, e.g., if the NDV of the second RS is very low. Thus, we would like this > decision to be cost-based. We could use a simple heuristic that would work > fine for most of the cases without introducing regressions for existing > cases, e.g., if NDV for partition column is less than estimated parallelism > in the second RS, do not execute deduplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23365) Put RS deduplication optimization under cost based decision
[
https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116191#comment-17116191
]
Hive QA commented on HIVE-23365:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003973/HIVE-23365.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22612/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22612/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:20:56.462
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22612/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:20:56.466
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 19:20:57.320
+ rm -rf ../yetus_PreCommit-HIVE-Build-22612
+ mkdir ../yetus_PreCommit-HIVE-Build-22612
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22612
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22612/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed: itests/src/test/resources/testconfiguration.properties:389
Falling back to three-way merge...
Applied patch to 'itests/src/test/resources/testconfiguration.properties' with
conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:330: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:348: trailing whitespace.
Reducer 2
/data/hiveptest/working/scratch/build.patch:396: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:406: trailing whitespace.
null sort order:
/data/hiveptest/working/scratch/build.patch:407: trailing whitespace.
sort order:
error: patch failed: itests/src/test/resources/testconfiguration.properties:389
Falling back to three-way merge...
Applied patch to 'itests/src/test/resources/testconfiguration.properties' with
conflicts.
U itests/src/test/resources/testconfiguration.properties
warning: squelched 13 whitespace errors
warning: 18 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22612
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003973 - PreCommit-HIVE-Build
> Put RS deduplication optimization under cost based decision
> ---
>
> Key: HIVE-23365
> URL: https://issues.apache.org/jira/browse/HIVE-23365
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch
>
>
> Currently, RS deduplication is always executed whenever it is semantically
> correct. However, it could be beneficial to
[jira] [Commented] (HIVE-23544) Add missing StageType enum values to thrift file after HIVE-23351
[
https://issues.apache.org/jira/browse/HIVE-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116160#comment-17116160
]
Hive QA commented on HIVE-23544:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003958/HIVE-23544.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17283 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22609/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22609/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22609/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003958 - PreCommit-HIVE-Build
> Add missing StageType enum values to thrift file after HIVE-23351
> -
>
> Key: HIVE-23544
> URL: https://issues.apache.org/jira/browse/HIVE-23544
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23544.01.patch, HIVE-23544.01.patch,
> HIVE-23544.01.patch
>
>
> HIVE-23351 adds references to new StageType enum values, but doesn't extend
> the thrift file, so a full build from scratch, including thrift generation,
> fails.
> {code}
> mvn clean install -DskipTests -Pthriftif -Dthrift.home=/usr/local
> {code}
> {code}
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:30 min
> [INFO] Finished at: 2020-05-24T23:32:13+02:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerDumpTask.java:[130,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_DUMP
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/AckTask.java:[53,21]
> cannot find symbol
> [ERROR] symbol: variable ACK
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR]
> /home/abstractdog/apache/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/RangerLoadTask.java:[147,21]
> cannot find symbol
> [ERROR] symbol: variable RANGER_LOAD
> [ERROR] location: class org.apache.hadoop.hive.ql.plan.api.StageType
> [ERROR] -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22942) Replace PTest with an alternative
[
https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116152#comment-17116152
]
Hive QA commented on HIVE-22942:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
38s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} llap-client: The patch generated 0 new + 5 unchanged
- 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} The patch kafka-handler passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m
44s{color} | {color:red} root: The patch generated 1 new + 238 unchanged - 2
fixed = 239 total (was 240) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 215
unchanged - 1 fixed = 216 total (was 216) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} The patch qtest passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
3s{color} | {color:red} patch/llap-client cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 10m
20s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} patch/kafka-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color}
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Attachment: (was: HIVE-23281.8.patch) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Open (was: Patch Available) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Patch Available (was: Open) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Attachment: HIVE-23281.8.patch > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23521) REPL: Optimise partition loading during bootstrap
[ https://issues.apache.org/jira/browse/HIVE-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23521: Attachment: HIVE-23521.2.patch > REPL: Optimise partition loading during bootstrap > - > > Key: HIVE-23521 > URL: https://issues.apache.org/jira/browse/HIVE-23521 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23521.1.patch, HIVE-23521.2.patch > > > When bootstrapping with large "REPL dump" with ~10K partitions, it starts > executing "addPartition" in sequential manner and takes very long time as it > communicates with HMS/registers partition etc for every call. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L399] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L165] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L210] > When bootstrap loading has to deal with DDL, it would be good to collate all > partitions in single call to HMS. This would help in reducing overall runtime. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[ https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23488: Attachment: HIVE-23488.2.patch > Optimise PartitionManagementTask::Msck::repair > -- > > Key: HIVE-23488 > URL: https://issues.apache.org/jira/browse/HIVE-23488 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23488.1.patch, HIVE-23488.2.patch, Screenshot > 2020-05-18 at 5.06.15 AM.png > > > Ends up fetching table information twice. > !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754! > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[
https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116255#comment-17116255
]
Hive QA commented on HIVE-23547:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003976/HIVE-23547.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22613/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22613/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22613/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-25 20:51:58.223
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22613/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 20:51:58.226
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9075e26 HIVE-23517: Update perf driver queries: q4 q11 q74
(Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-25 20:51:59.279
+ rm -rf ../yetus_PreCommit-HIVE-Build-22613
+ mkdir ../yetus_PreCommit-HIVE-Build-22613
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22613
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22613/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:52: trailing whitespace.
echo "test name should contain only letters, numbers and '_' characters,
but in row $row: '$line'"
warning: 1 line adds whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc5059326922909985590.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc5059326922909985590.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc1260930769238641935.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
[jira] [Updated] (HIVE-23535) Bump Minimum Required Version of Maven to 3.0.5
[
https://issues.apache.org/jira/browse/HIVE-23535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-23535:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, [~belugabehr]
> Bump Minimum Required Version of Maven to 3.0.5
> ---
>
> Key: HIVE-23535
> URL: https://issues.apache.org/jira/browse/HIVE-23535
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23535.1.patch
>
>
> {code:xml|title=pom.xml}
>
> 2.2.1
>
> {code}
> Time to upgrade to 3.x
> {quote}In Maven 3, use Maven Enforcer Plugin's requireMavenVersion rule, or
> other rules to check other aspects.
> {quote}
> [https://maven.apache.org/pom.html#prerequisites]
>
> We get the Enforcer Plugin from the Apache Parent POM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116342#comment-17116342
]
Hive QA commented on HIVE-23468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003987/HIVE-23468.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17284 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22616/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22616/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22616/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003987 - PreCommit-HIVE-Build
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-21971:
Attachment: HIVE-21971.5.patch
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch, HIVE-21971.5.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[ https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21624: Attachment: HIVE-21624.3.patch > LLAP: Cpu metrics at thread level is broken > --- > > Key: HIVE-21624 > URL: https://issues.apache.org/jira/browse/HIVE-21624 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch, > HIVE-21624.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu > metrics when available. At some point, the thread name which the metrics > publisher looks for has changed causing no metrics to be published for these > counters. > The above counters looks for thread with name starting with > "ContainerExecutor" but the llap task executor thread got changed to > "Task-Executor" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[ https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21624: Status: Open (was: Patch Available) > LLAP: Cpu metrics at thread level is broken > --- > > Key: HIVE-21624 > URL: https://issues.apache.org/jira/browse/HIVE-21624 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch, > HIVE-21624.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu > metrics when available. At some point, the thread name which the metrics > publisher looks for has changed causing no metrics to be published for these > counters. > The above counters looks for thread with name starting with > "ContainerExecutor" but the llap task executor thread got changed to > "Task-Executor" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[ https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21624: Status: Patch Available (was: Open) > LLAP: Cpu metrics at thread level is broken > --- > > Key: HIVE-21624 > URL: https://issues.apache.org/jira/browse/HIVE-21624 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch, > HIVE-21624.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu > metrics when available. At some point, the thread name which the metrics > publisher looks for has changed causing no metrics to be published for these > counters. > The above counters looks for thread with name starting with > "ContainerExecutor" but the llap task executor thread got changed to > "Task-Executor" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116297#comment-17116297
]
Ashutosh Chauhan commented on HIVE-23266:
-
can you please create a Review Board for it?
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.10.patch,
> HIVE-23266.11.patch, HIVE-23266.2.patch, HIVE-23266.2.patch,
> HIVE-23266.3.patch, HIVE-23266.4.patch, HIVE-23266.5.patch,
> HIVE-23266.6.patch, HIVE-23266.7.patch, HIVE-23266.8.patch, HIVE-23266.9.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23175) Skip serializing hadoop and tez config on HS side
[ https://issues.apache.org/jira/browse/HIVE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116304#comment-17116304 ] Ashutosh Chauhan commented on HIVE-23175: - [~mustafaiman] would you like to rebase your patch? Two new methods introduced in TezUtils in TEZ-4137 are static methods and we can duplicate those in Hive temporarily while waiting for a new Tez release. > Skip serializing hadoop and tez config on HS side > - > > Key: HIVE-23175 > URL: https://issues.apache.org/jira/browse/HIVE-23175 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23175.1.patch, HIVE-23175.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer spends a lot of time serializing configuration objects. We can > skip putting hadoop and tez config xml files in payload assuming that the > configs are the same on both HS and Task side. This depends on Tez to load > local xml configs when creating config objects > [https://issues.apache.org/jira/browse/TEZ-4137] > Ideally we should be able to skip hive-site.xml too. However, if we skip > hive-site.xml at that stage, then we make wrong choices at tez dag build > stage due to missing configs. > In the ideal version of this, we should not be both looking up configs and > putting new configs from and to the same config object at DAG and Vertex > build phases. Instead we should be looking up from a HS2's HiveConf object > and writing to a brand new JobConf for each vertex. That way we would not > have any unnecessary item in the jobconf for any vertex. However Dag and > Vertex build stages (TezTask#build) and a lot of other components called from > there treat a single config object both the source of HS2 side config and the > target JobConf that they are putting vertex level options into. It is very > hard to separate these concerns now. > With this patch, we are reducing the size of JobConf (per vertex) by ~65%. It > should improve the transmit latency. However, most significant gains are at > CPU time while compressing job configs as the config objects are much smaller > now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116337#comment-17116337
]
Hive QA commented on HIVE-23468:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22616/dev-support/hive-personality.sh
|
| git revision | master / 947b7a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22616/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[ https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23547: -- Attachment: HIVE-23547.03.patch > Enforce testconfiguration.properties file format and alphabetical order > --- > > Key: HIVE-23547 > URL: https://issues.apache.org/jira/browse/HIVE-23547 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > Attachments: HIVE-23547.01.patch, HIVE-23547.02.patch, > HIVE-23547.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23547) Enforce testconfiguration.properties file format and alphabetical order
[
https://issues.apache.org/jira/browse/HIVE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116256#comment-17116256
]
Hive QA commented on HIVE-23547:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003976/HIVE-23547.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22614/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22614/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/13003976/HIVE-23547.03.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003976 - PreCommit-HIVE-Build
> Enforce testconfiguration.properties file format and alphabetical order
> ---
>
> Key: HIVE-23547
> URL: https://issues.apache.org/jira/browse/HIVE-23547
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23547.01.patch, HIVE-23547.02.patch,
> HIVE-23547.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Patch Available (was: Open) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23487) Optimise PartitionManagementTask
[ https://issues.apache.org/jira/browse/HIVE-23487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23487: Attachment: HIVE-23487.2.patch > Optimise PartitionManagementTask > > > Key: HIVE-23487 > URL: https://issues.apache.org/jira/browse/HIVE-23487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23487.1.patch, HIVE-23487.2.patch, Screenshot > 2020-05-18 at 4.19.48 AM.png > > > Msck.init for every table takes more time than the actual table repair. This > was observed on a system which had lots of DB and tables. > > !Screenshot 2020-05-18 at 4.19.48 AM.png|width=1014,height=732! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116311#comment-17116311
]
Hive QA commented on HIVE-23281:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 6 new + 586 unchanged - 1 fixed = 592 total (was 587) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22615/dev-support/hive-personality.sh
|
| git revision | master / 9075e26 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22615/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22615/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22615/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22615/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to
> DB for ACID tables
> --
>
> Key: HIVE-23281
> URL: https://issues.apache.org/jira/browse/HIVE-23281
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch,
> HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch,
> HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch,
> image-2020-04-23-13-56-17-210.png
>
>
>
[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116315#comment-17116315
]
Hive QA commented on HIVE-23281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003986/HIVE-23281.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17283 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_09_nonpart_noncompat_serdeparam]
(batchId=6)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22615/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22615/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22615/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003986 - PreCommit-HIVE-Build
> ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to
> DB for ACID tables
> --
>
> Key: HIVE-23281
> URL: https://issues.apache.org/jira/browse/HIVE-23281
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch,
> HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch,
> HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch,
> image-2020-04-23-13-56-17-210.png
>
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980]
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982]
>
> SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be
> good to check for transactional tables and get rid these calls in table
> lookups.
>
> This should help in reducing DB calls.
> !image-2020-04-23-13-56-17-210.png|width=669,height=485!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Attachment: HIVE-23281.9.patch > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > HIVE-23281.9.patch, image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Patch Available (was: Open) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > HIVE-23281.9.patch, image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Open (was: Patch Available) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > HIVE-23281.9.patch, image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Attachment: HIVE-23281.8.patch > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, HIVE-23281.8.patch, > image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23281: Status: Open (was: Patch Available) > ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to > DB for ACID tables > -- > > Key: HIVE-23281 > URL: https://issues.apache.org/jira/browse/HIVE-23281 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23281.1.patch, HIVE-23281.2.patch, > HIVE-23281.3.patch, HIVE-23281.4.patch, HIVE-23281.5.patch, > HIVE-23281.6.patch, HIVE-23281.7.patch, image-2020-04-23-13-56-17-210.png > > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980] > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1982] > > SkewInfo, bucketCols, ordering etc are not needed for ACID tables. It may be > good to check for transactional tables and get rid these calls in table > lookups. > > This should help in reducing DB calls. > !image-2020-04-23-13-56-17-210.png|width=669,height=485! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23468: Attachment: HIVE-23468.4.patch > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23536) Provide an option to skip stats generation for major compaction
[ https://issues.apache.org/jira/browse/HIVE-23536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23536: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Peter! > Provide an option to skip stats generation for major compaction > --- > > Key: HIVE-23536 > URL: https://issues.apache.org/jira/browse/HIVE-23536 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23536.02.patch, HIVE-23536.03.patch, > HIVE-23536.patch > > > Currently major MR compaction is regenerates stats every time if the column > stats table contains some data. Some configurations do not use stats but > because of historical reasons the column stats table can still contain some > data. > We should provide a possibility to skip stats generation in these cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23536) Provide an option to skip stats generation for major compaction
[ https://issues.apache.org/jira/browse/HIVE-23536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116301#comment-17116301 ] Ashutosh Chauhan commented on HIVE-23536: - +1 > Provide an option to skip stats generation for major compaction > --- > > Key: HIVE-23536 > URL: https://issues.apache.org/jira/browse/HIVE-23536 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23536.02.patch, HIVE-23536.03.patch, > HIVE-23536.patch > > > Currently major MR compaction is regenerates stats every time if the column > stats table contains some data. Some configurations do not use stats but > because of historical reasons the column stats table can still contain some > data. > We should provide a possibility to skip stats generation in these cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23546) Skip authorization when user is a superuser
[ https://issues.apache.org/jira/browse/HIVE-23546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116361#comment-17116361 ] Zhihua Deng commented on HIVE-23546: [~navis], [~mgergely], [~pgaref] Could you take some time to look at this? > Skip authorization when user is a superuser > --- > > Key: HIVE-23546 > URL: https://issues.apache.org/jira/browse/HIVE-23546 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23546.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If the current user is a superuser, there is no need to do authorization. > This can speed up queries, especially for those ddl queries. For example, the > superuser add partitions when the external data is ready, or show partitions > to check whether it OK to take the work flow one step further in a busy hive > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116367#comment-17116367
]
Hive QA commented on HIVE-21971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003989/HIVE-21971.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17286 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22617/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22617/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22617/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003989 - PreCommit-HIVE-Build
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch, HIVE-21971.5.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21624) LLAP: Cpu metrics at thread level is broken
[
https://issues.apache.org/jira/browse/HIVE-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116400#comment-17116400
]
Hive QA commented on HIVE-21624:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003988/HIVE-21624.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17284 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=130)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22618/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22618/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22618/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003988 - PreCommit-HIVE-Build
> LLAP: Cpu metrics at thread level is broken
> ---
>
> Key: HIVE-21624
> URL: https://issues.apache.org/jira/browse/HIVE-21624
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21624.1.patch, HIVE-21624.2.patch,
> HIVE-21624.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ExecutorThreadCPUTime and ExecutorThreadUserTime relies on thread mx bean cpu
> metrics when available. At some point, the thread name which the metrics
> publisher looks for has changed causing no metrics to be published for these
> counters.
> The above counters looks for thread with name starting with
> "ContainerExecutor" but the llap task executor thread got changed to
> "Task-Executor"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23521) REPL: Optimise partition loading during bootstrap
[
https://issues.apache.org/jira/browse/HIVE-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116442#comment-17116442
]
Hive QA commented on HIVE-23521:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
34s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22620/dev-support/hive-personality.sh
|
| git revision | master / 947b7a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22620/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> REPL: Optimise partition loading during bootstrap
> -
>
> Key: HIVE-23521
> URL: https://issues.apache.org/jira/browse/HIVE-23521
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23521.1.patch, HIVE-23521.2.patch
>
>
> When bootstrapping with large "REPL dump" with ~10K partitions, it starts
> executing "addPartition" in sequential manner and takes very long time as it
> communicates with HMS/registers partition etc for every call.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L399]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L165]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L210]
> When bootstrap loading has to deal with DDL, it would be good to collate all
> partitions in single call to HMS. This would help in reducing overall runtime.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
