[jira] [Commented] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105735#comment-17105735 ] Gopal Vijayaraghavan commented on HIVE-23453: - LGTM - +1 tests pending. Looks like we have a patch conflict upcoming due to HIVE-23423 > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Attachments: HIVE-23453.patch > > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105833#comment-17105833
]
Hive QA commented on HIVE-23453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} llap-server in master has 87 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} llap-server: The patch generated 0 new + 3 unchanged
- 4 fixed = 3 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22296/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22296/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> IntelliJ compile errors in StaticPermanentFunctionChecker and
> TestVectorGroupByOperator
> ---
>
> Key: HIVE-23453
> URL: https://issues.apache.org/jira/browse/HIVE-23453
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91
> jdk 1.8.0_251
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23453.patch
>
>
> The following errors appear when compiling the code using IntelliJ:
> TestVectorGroupByOperator: Error:(89, 32) java: package
> com.sun.tools.javac.util does not exist
> StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does
> not exist
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105850#comment-17105850
]
Hive QA commented on HIVE-23453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002752/HIVE-23453.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17266 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics
(batchId=219)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22296/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22296/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002752 - PreCommit-HIVE-Build
> IntelliJ compile errors in StaticPermanentFunctionChecker and
> TestVectorGroupByOperator
> ---
>
> Key: HIVE-23453
> URL: https://issues.apache.org/jira/browse/HIVE-23453
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91
> jdk 1.8.0_251
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23453.patch
>
>
> The following errors appear when compiling the code using IntelliJ:
> TestVectorGroupByOperator: Error:(89, 32) java: package
> com.sun.tools.javac.util does not exist
> StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does
> not exist
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105847#comment-17105847 ] Ashutosh Chauhan commented on HIVE-23449: - Left a comment on RB . Also, some of failures seems related to patch : TestContainerRunnerImpl.testRegisterDag() > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[ https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-23423: Attachment: HIVE-23423.1.patch > Hash aggregation is always disabled in vectorized execution of grouping set > queries > --- > > Key: HIVE-23423 > URL: https://issues.apache.org/jira/browse/HIVE-23423 > Project: Hive > Issue Type: Bug > Components: Hive, Operators, Query Processor >Affects Versions: 4.0.0 >Reporter: Nita Dembla >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Performance > Attachments: HIVE-23423.1.patch, HIVE-23423.WIP.patch > > > https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with > disabling hash aggregation on grouping set queries. Need a fix for > VectorGroupbyOperator operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23017) Use Inherited Logger in Tasks
[
https://issues.apache.org/jira/browse/HIVE-23017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23017:
--
Attachment: HIVE-23017.4.patch
> Use Inherited Logger in Tasks
> -
>
> Key: HIVE-23017
> URL: https://issues.apache.org/jira/browse/HIVE-23017
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23017.1.patch, HIVE-23017.2.patch,
> HIVE-23017.3.patch, HIVE-23017.4.patch
>
>
> {code:java|title=FetchTask}
> public class FetchTask extends Task implements Serializable {
> private static final long serialVersionUID = 1L;
> private int maxRows = 100;
> private FetchOperator fetch;
> private ListSinkOperator sink;
> private int totalRows;
> private static transient final Logger LOG =
> LoggerFactory.getLogger(FetchTask.class);
> JobConf job = null;
> {code}
> The Tasks class is base class for many other. Cleaner to simply make the
> {{Logger}} {{protected}} and usable by other classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105819#comment-17105819
]
Hive QA commented on HIVE-23099:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002753/HIVE-23099.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17265 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.streaming.TestStreaming.testConcurrentTransactionBatchCommits
(batchId=149)
org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth
(batchId=219)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22295/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22295/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002753 - PreCommit-HIVE-Build
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch, HIVE-23099.5.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.2.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[
https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105862#comment-17105862
]
Hive QA commented on HIVE-22675:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
29s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 31 unchanged - 1 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22297/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22297/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22297/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in hive-standalone-metastore Package
> ---
>
> Key: HIVE-22675
> URL: https://issues.apache.org/jira/browse/HIVE-22675
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch,
> HIVE-22675.3.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105873#comment-17105873 ] Ashutosh Chauhan commented on HIVE-23451: - How is totalFiles used? Can you explain how is this suppose to work? I wonder alternative approach could be following. This deletion is invoked in createBucketForFileIdx(). createBucketFiles() calls the first one, so it will be called twice. I wonder whether we shall just remove this if() call altogether at: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L796 since invocation from https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L790 will already result in call at https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826 for all indices. I dont see any reason to treat index 0 especially as its done currently. > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23017) Use Inherited Logger in Tasks
[
https://issues.apache.org/jira/browse/HIVE-23017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105886#comment-17105886
]
Hive QA commented on HIVE-23017:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002764/HIVE-23017.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22298/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22298/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-13 01:55:26.725
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22298/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 01:55:26.728
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2ff6370 HIVE-23053: Clean Up Stats Mergers (David Mollitor,
reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2ff6370 HIVE-23053: Clean Up Stats Mergers (David Mollitor,
reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 01:55:27.674
+ rm -rf ../yetus_PreCommit-HIVE-Build-22298
+ mkdir ../yetus_PreCommit-HIVE-Build-22298
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22298
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22298/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLTask.java: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainSQRewriteTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplTxnTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/DirCopyTask.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintenanceTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java: does
not exist in index
error:
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105787#comment-17105787
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956626/HIVE-21052.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22294/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22294/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22294/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-12 22:17:49.335
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22294/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-12 22:17:49.339
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2ff6370 HIVE-23053: Clean Up Stats Mergers (David Mollitor,
reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2ff6370 HIVE-23053: Clean Up Stats Mergers (David Mollitor,
reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-12 22:17:50.384
+ rm -rf ../yetus_PreCommit-HIVE-Build-22294
+ mkdir ../yetus_PreCommit-HIVE-Build-22294
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22294
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22294/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java:142
Falling back to three-way merge...
Applied patch to
'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java'
cleanly.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:27
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java:25
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:96
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java' with
conflicts.
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java:1750
Falling back to three-way merge...
Applied patch to
'ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java' with
conflicts.
error: patch failed:
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java:40
Falling back to three-way merge...
Applied patch to
'shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java:117
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:83
Falling back to three-way merge...
Applied patch to
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105863#comment-17105863 ] John Sherman commented on HIVE-23354: - I've adjusted the patch to check for speculative execution and throw an exception. After discussion with other Hive contributors/committers, MR is not as widely used with newer version of Hive and speculative execution with Tez seems to be marked as unstable. Especially since this code as it currently exists is also broken for speculative execution. A follow up Jira will be made once this patch gets reviewed to look into supporting speculative execution in this path. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[ https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] De Li updated HIVE-23452: - Description: Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by Tez and it works when query with MR. (was: Report Exception when querying based on two JDBC StorageHandler table with Tez, it seems Tez found incorrect driver. It just happen on Tez and it works on MR.) Summary: Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez (was: Report Exception when querying based on two JDBC StorageHandler table with Tez) > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez > --- > > Key: HIVE-23452 > URL: https://issues.apache.org/jira/browse/HIVE-23452 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: De Li >Priority: Major > > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by > Tez and it works when query with MR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23449: Attachment: HIVE-23449.2.patch > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, Screenshot > 2020-05-12 at 1.09.35 PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23455) Improve error message for external orc table
[
https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105905#comment-17105905
]
Hive QA commented on HIVE-23455:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22299/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22299/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve error message for external orc table
>
>
> Key: HIVE-23455
> URL: https://issues.apache.org/jira/browse/HIVE-23455
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23455.patch, acid_table_as_external.q,
> acid_table_as_external.q.out
>
>
> Since there is no schema validation for external tables, users may face
> various errors if their orc data and external table schema does not match. If
> orc schema has fewer columns than projection OrcEncodedDataConsumer may
> receive an incomplete TypeDescription array which will manifest itself as
> NullPointerException later.
> We can at least verify that OrcEncodedDataConsumer gets enough
> TypeDescriptions. If assertion fails, user sees there is something wrong with
> the schema and hopefully resolves the problem quickly. If there are enough
> columns in the file but the schema of the query does not match, user
> generally sees a ClassCastException. If there are enough columns and types
> accidentally match, there is nothing we can do as this is an external table.
> We have seen this when trying to use a managed table as external table
> location. Although user facing schemas are the same, managed table has acid
> related metadata. I am adding a q file demonstrating NullPointerException
> with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added
> this to precommit tests as it is hard to assert the exception message from
> mini driver framework and effectively it is just changing the error.
[jira] [Commented] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105925#comment-17105925 ] Ashutosh Chauhan commented on HIVE-23449: - +1 pending tests. > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, Screenshot > 2020-05-12 at 1.09.35 PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105767#comment-17105767
]
Hive QA commented on HIVE-23444:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 1 new + 167 unchanged - 0
fixed = 168 total (was 167) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22293/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22293/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22293/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
>
[jira] [Assigned] (HIVE-23455) Improve error message for external orc table
[ https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman reassigned HIVE-23455: --- > Improve error message for external orc table > > > Key: HIVE-23455 > URL: https://issues.apache.org/jira/browse/HIVE-23455 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > > Since there is no schema validation for external tables, users may face > various errors if their orc data and external table schema does not match. If > orc schema has fewer columns than projection OrcEncodedDataConsumer may > receive an incomplete TypeDescription array which will manifest itself as > NullPointerException later. > We can at least verify that OrcEncodedDataConsumer gets enough > TypeDescriptions. If assertion fails, user sees there is something wrong with > the schema and hopefully resolves the problem quickly. If there are enough > columns in the file but the schema of the query does not match, user > generally sees a ClassCastException. If there are enough columns and types > accidentally match, there is nothing we can do as this is an external table. > We have seen this when trying to use a managed table as external table > location. Although user facing schemas are the same, managed table has acid > related metadata. I am adding a q file demonstrating NullPointerException > with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added > this to precommit tests as it is hard to assert the exception message from > mini driver framework and effectively it is just changing the error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23456) Upgrade Calcite version to 1.23.0
[ https://issues.apache.org/jira/browse/HIVE-23456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis reassigned HIVE-23456: -- > Upgrade Calcite version to 1.23.0 > - > > Key: HIVE-23456 > URL: https://issues.apache.org/jira/browse/HIVE-23456 > Project: Hive > Issue Type: Task >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105776#comment-17105776 ] Rajesh Balamohan commented on HIVE-23449: - RB: https://reviews.apache.org/r/72503/diff/1#index_header > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105786#comment-17105786
]
Hive QA commented on HIVE-23444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002750/HIVE-23444.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 17267 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers1
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerShortQueryElapsedTime
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=218)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsNoKill
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22293/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22293/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22293/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002750 - PreCommit-HIVE-Build
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
>
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105726#comment-17105726
]
Hive QA commented on HIVE-23432:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 2
fixed = 7 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22292/dev-support/hive-personality.sh
|
| git revision | master / 57c1593 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22292/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105740#comment-17105740
]
Hive QA commented on HIVE-23432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002744/HIVE-23432.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=78)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22292/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22292/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22292/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002744 - PreCommit-HIVE-Build
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[
https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105879#comment-17105879
]
Hive QA commented on HIVE-22675:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002763/HIVE-22675.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17266 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.security.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
(batchId=223)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testReplTimeouts
(batchId=244)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22297/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22297/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002763 - PreCommit-HIVE-Build
> Replace Base64 in hive-standalone-metastore Package
> ---
>
> Key: HIVE-22675
> URL: https://issues.apache.org/jira/browse/HIVE-22675
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch,
> HIVE-22675.3.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23455) Improve error message for external orc table
[ https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman updated HIVE-23455: Attachment: HIVE-23455.patch Status: Patch Available (was: Open) > Improve error message for external orc table > > > Key: HIVE-23455 > URL: https://issues.apache.org/jira/browse/HIVE-23455 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23455.patch, acid_table_as_external.q, > acid_table_as_external.q.out > > > Since there is no schema validation for external tables, users may face > various errors if their orc data and external table schema does not match. If > orc schema has fewer columns than projection OrcEncodedDataConsumer may > receive an incomplete TypeDescription array which will manifest itself as > NullPointerException later. > We can at least verify that OrcEncodedDataConsumer gets enough > TypeDescriptions. If assertion fails, user sees there is something wrong with > the schema and hopefully resolves the problem quickly. If there are enough > columns in the file but the schema of the query does not match, user > generally sees a ClassCastException. If there are enough columns and types > accidentally match, there is nothing we can do as this is an external table. > We have seen this when trying to use a managed table as external table > location. Although user facing schemas are the same, managed table has acid > related metadata. I am adding a q file demonstrating NullPointerException > with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added > this to precommit tests as it is hard to assert the exception message from > mini driver framework and effectively it is just changing the error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23455) Improve error message for external orc table
[ https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman updated HIVE-23455: Attachment: acid_table_as_external.q acid_table_as_external.q.out > Improve error message for external orc table > > > Key: HIVE-23455 > URL: https://issues.apache.org/jira/browse/HIVE-23455 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23455.patch, acid_table_as_external.q, > acid_table_as_external.q.out > > > Since there is no schema validation for external tables, users may face > various errors if their orc data and external table schema does not match. If > orc schema has fewer columns than projection OrcEncodedDataConsumer may > receive an incomplete TypeDescription array which will manifest itself as > NullPointerException later. > We can at least verify that OrcEncodedDataConsumer gets enough > TypeDescriptions. If assertion fails, user sees there is something wrong with > the schema and hopefully resolves the problem quickly. If there are enough > columns in the file but the schema of the query does not match, user > generally sees a ClassCastException. If there are enough columns and types > accidentally match, there is nothing we can do as this is an external table. > We have seen this when trying to use a managed table as external table > location. Although user facing schemas are the same, managed table has acid > related metadata. I am adding a q file demonstrating NullPointerException > with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added > this to precommit tests as it is hard to assert the exception message from > mini driver framework and effectively it is just changing the error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105795#comment-17105795
]
Hive QA commented on HIVE-23099:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} service: The patch generated 1 new + 57 unchanged - 2
fixed = 58 total (was 59) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22295/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22295/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22295/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch, HIVE-23099.5.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105809#comment-17105809
]
Gopal Vijayaraghavan commented on HIVE-23443:
-
bq. 1) If guaranteed or finishable, the task should not be in pre-emption queue
This is very suspiciously like the pre-WLM model (which we know works without
deadlocks), but there's no way to fix pool based preemption if a speculative
finishable takes over a whole cluster.
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23443.1.patch
>
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Attachment: HIVE-23432.03.patch Status: Patch Available (was: In Progress) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Status: In Progress (was: Patch Available) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23455) Improve error message for external orc table
[
https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105928#comment-17105928
]
Ashutosh Chauhan commented on HIVE-23455:
-
Instead of writing a new loop, can this check be included while typeDescription
is getting populated. Essentially
{code}
result[indexInBatchCols] = children.get(columnNumber);
if result[indexInBatchCols] throw AssertionException();
{code}
For tests we also have TestNegativeCliDriver which can be used for such
negative scenario.
> Improve error message for external orc table
>
>
> Key: HIVE-23455
> URL: https://issues.apache.org/jira/browse/HIVE-23455
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23455.patch, acid_table_as_external.q,
> acid_table_as_external.q.out
>
>
> Since there is no schema validation for external tables, users may face
> various errors if their orc data and external table schema does not match. If
> orc schema has fewer columns than projection OrcEncodedDataConsumer may
> receive an incomplete TypeDescription array which will manifest itself as
> NullPointerException later.
> We can at least verify that OrcEncodedDataConsumer gets enough
> TypeDescriptions. If assertion fails, user sees there is something wrong with
> the schema and hopefully resolves the problem quickly. If there are enough
> columns in the file but the schema of the query does not match, user
> generally sees a ClassCastException. If there are enough columns and types
> accidentally match, there is nothing we can do as this is an external table.
> We have seen this when trying to use a managed table as external table
> location. Although user facing schemas are the same, managed table has acid
> related metadata. I am adding a q file demonstrating NullPointerException
> with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added
> this to precommit tests as it is hard to assert the exception message from
> mini driver framework and effectively it is just changing the error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23455) Improve error message for external orc table
[
https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105927#comment-17105927
]
Hive QA commented on HIVE-23455:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002768/HIVE-23455.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17266 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
(batchId=167)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22299/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22299/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002768 - PreCommit-HIVE-Build
> Improve error message for external orc table
>
>
> Key: HIVE-23455
> URL: https://issues.apache.org/jira/browse/HIVE-23455
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23455.patch, acid_table_as_external.q,
> acid_table_as_external.q.out
>
>
> Since there is no schema validation for external tables, users may face
> various errors if their orc data and external table schema does not match. If
> orc schema has fewer columns than projection OrcEncodedDataConsumer may
> receive an incomplete TypeDescription array which will manifest itself as
> NullPointerException later.
> We can at least verify that OrcEncodedDataConsumer gets enough
> TypeDescriptions. If assertion fails, user sees there is something wrong with
> the schema and hopefully resolves the problem quickly. If there are enough
> columns in the file but the schema of the query does not match, user
> generally sees a ClassCastException. If there are enough columns and types
> accidentally match, there is nothing we can do as this is an external table.
> We have seen this when trying to use a managed table as external table
> location. Although user facing schemas are the same, managed table has acid
> related metadata. I am adding a q file demonstrating NullPointerException
> with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added
> this to precommit tests as it is hard to assert the exception message from
> mini driver framework and effectively it is just changing the error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[
https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105209#comment-17105209
]
Hive QA commented on HIVE-23409:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002676/HIVE-23409.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17267 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22279/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22279/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002676 - PreCommit-HIVE-Build
> If TezSession application reopen fails for Timeline service down, default
> TezSession from SessionPool is closed after a retry
> -
>
> Key: HIVE-23409
> URL: https://issues.apache.org/jira/browse/HIVE-23409
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-23409.1.patch, HIVE-23409.patch
>
>
> we are closing a default session from TezSessionPool at here.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589]
> If all the sessions in a pool are destroyed, queries wait indefinitely at
> TezSessionPool.getSession until HS2 restarts after other service recoveries.
> [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:())
> - We are closing a default session because of retry failure.
> It's better if we allow retry & fail than hung.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23438) Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
[
https://issues.apache.org/jira/browse/HIVE-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-23438:
---
Attachment: HIVE-23438.branch-2.3.patch
Status: Patch Available (was: Open)
> Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
> --
>
> Key: HIVE-23438
> URL: https://issues.apache.org/jira/browse/HIVE-23438
> Project: Hive
> Issue Type: Bug
> Components: SQL, Tez
>Affects Versions: 2.3.4
>Reporter: 范宜臻
>Priority: Major
> Attachments: HIVE-23438.branch-2.3.patch
>
>
> *Run Test in Patch File*
> {code:java}
> mvn test -Dtest=TestMiniTezCliDriver -Dqfile=hybridgrace_hashjoin_2.q{code}
> *Manual Reproduce*
> *STEP 1. Create test data(q_test_init_tez.sql)*
> {code:java}
> //create table src1
> CREATE TABLE src1 (key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv3.txt" INTO TABLE src1;
> //create table src2
> CREATE TABLE src2(key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv11.txt" OVERWRITE INTO
> TABLE src2;
> //create table srcpart
> CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT
> 'default')
> PARTITIONED BY (ds STRING, hr STRING)
> STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="12");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="12");{code}
> *STEP 2. Run query*
> {code:java}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.noconditionaltask=true;
> set hive.auto.convert.join.noconditionaltask.size=1000;
> set hive.cbo.enable=false;
> set hive.mapjoin.hybridgrace.hashtable=true;
> select *
> from
> (
> select key from src1 group by key
> ) x
> left join src2 z on x.key = z.key
> join
> (
> select key from srcpart y group by key
> ) y on y.key = x.key;
> {code}
> *EXPECTED RESULT***
>
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 238 NULLNULL238
> 369 NULLNULL369
> 406 406 1val_4061 406
> 273 273 1val_2731 273
> 98NULLNULL98
> 213 213 1val_2131 213
> 255 NULLNULL255
> 401 401 1val_4011 401
> 278 NULLNULL278
> 6666 11val_6611 66
> 224 NULLNULL224
> 311 NULLNULL311
> {code}
>
> *ACTUAL RESULT*
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 213 213 1val_2131 213
> 238 NULLNULL238
> 273 273 1val_2731 273
> 369 NULLNULL369
> 406 406 1val_4061 406
> 98NULLNULL98
> 401 401 1val_4011 401
> 6666 11val_6611 66
> {code}
>
> *ROOT CAUSE*
> src1 left join src2, src1 is big table and src2 is small table. Join result
> between big table row and the corresponding hashtable maybe NO_MATCH state,
> however, these NO_MATCH rows is needed because LEFT OUTER JOIN.
> In addition, these big table rows will not spilled into matchfile related to
> this hashtable on disk because only SPILL state can use `spillBigTableRow`.
> Then, these big table rows will be spilled into matchfile in hashtables of
> table `srcpart`(second small table)
> Finally, when reProcessBigTable, big table rows in matchfile are only read
> from `firstSmallTable`, some datum are missing.
>
> *WORKAROUND*
> configure firstSmallTable in completeInitializationOp and only spill big
> table row into firstSmallTable when spill matchfile.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23438) Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
[
https://issues.apache.org/jira/browse/HIVE-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-23438:
---
Status: Open (was: Patch Available)
> Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
> --
>
> Key: HIVE-23438
> URL: https://issues.apache.org/jira/browse/HIVE-23438
> Project: Hive
> Issue Type: Bug
> Components: SQL, Tez
>Affects Versions: 2.3.4
>Reporter: 范宜臻
>Priority: Major
> Attachments: HIVE-23438.patch
>
>
> *Run Test in Patch File*
> {code:java}
> mvn test -Dtest=TestMiniTezCliDriver -Dqfile=hybridgrace_hashjoin_2.q{code}
> *Manual Reproduce*
> *STEP 1. Create test data(q_test_init_tez.sql)*
> {code:java}
> //create table src1
> CREATE TABLE src1 (key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv3.txt" INTO TABLE src1;
> //create table src2
> CREATE TABLE src2(key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv11.txt" OVERWRITE INTO
> TABLE src2;
> //create table srcpart
> CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT
> 'default')
> PARTITIONED BY (ds STRING, hr STRING)
> STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="12");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="12");{code}
> *STEP 2. Run query*
> {code:java}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.noconditionaltask=true;
> set hive.auto.convert.join.noconditionaltask.size=1000;
> set hive.cbo.enable=false;
> set hive.mapjoin.hybridgrace.hashtable=true;
> select *
> from
> (
> select key from src1 group by key
> ) x
> left join src2 z on x.key = z.key
> join
> (
> select key from srcpart y group by key
> ) y on y.key = x.key;
> {code}
> *EXPECTED RESULT***
>
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 238 NULLNULL238
> 369 NULLNULL369
> 406 406 1val_4061 406
> 273 273 1val_2731 273
> 98NULLNULL98
> 213 213 1val_2131 213
> 255 NULLNULL255
> 401 401 1val_4011 401
> 278 NULLNULL278
> 6666 11val_6611 66
> 224 NULLNULL224
> 311 NULLNULL311
> {code}
>
> *ACTUAL RESULT*
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 213 213 1val_2131 213
> 238 NULLNULL238
> 273 273 1val_2731 273
> 369 NULLNULL369
> 406 406 1val_4061 406
> 98NULLNULL98
> 401 401 1val_4011 401
> 6666 11val_6611 66
> {code}
>
> *ROOT CAUSE*
> src1 left join src2, src1 is big table and src2 is small table. Join result
> between big table row and the corresponding hashtable maybe NO_MATCH state,
> however, these NO_MATCH rows is needed because LEFT OUTER JOIN.
> In addition, these big table rows will not spilled into matchfile related to
> this hashtable on disk because only SPILL state can use `spillBigTableRow`.
> Then, these big table rows will be spilled into matchfile in hashtables of
> table `srcpart`(second small table)
> Finally, when reProcessBigTable, big table rows in matchfile are only read
> from `firstSmallTable`, some datum are missing.
>
> *WORKAROUND*
> configure firstSmallTable in completeInitializationOp and only spill big
> table row into firstSmallTable when spill matchfile.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23438) Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
[
https://issues.apache.org/jira/browse/HIVE-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-23438:
---
Attachment: (was: HIVE-23438.patch)
> Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
> --
>
> Key: HIVE-23438
> URL: https://issues.apache.org/jira/browse/HIVE-23438
> Project: Hive
> Issue Type: Bug
> Components: SQL, Tez
>Affects Versions: 2.3.4
>Reporter: 范宜臻
>Priority: Major
>
> *Run Test in Patch File*
> {code:java}
> mvn test -Dtest=TestMiniTezCliDriver -Dqfile=hybridgrace_hashjoin_2.q{code}
> *Manual Reproduce*
> *STEP 1. Create test data(q_test_init_tez.sql)*
> {code:java}
> //create table src1
> CREATE TABLE src1 (key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv3.txt" INTO TABLE src1;
> //create table src2
> CREATE TABLE src2(key STRING COMMENT 'default', value STRING COMMENT
> 'default') STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv11.txt" OVERWRITE INTO
> TABLE src2;
> //create table srcpart
> CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT
> 'default')
> PARTITIONED BY (ds STRING, hr STRING)
> STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-08", hr="12");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11");
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt"
> OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="12");{code}
> *STEP 2. Run query*
> {code:java}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.noconditionaltask=true;
> set hive.auto.convert.join.noconditionaltask.size=1000;
> set hive.cbo.enable=false;
> set hive.mapjoin.hybridgrace.hashtable=true;
> select *
> from
> (
> select key from src1 group by key
> ) x
> left join src2 z on x.key = z.key
> join
> (
> select key from srcpart y group by key
> ) y on y.key = x.key;
> {code}
> *EXPECTED RESULT***
>
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 238 NULLNULL238
> 369 NULLNULL369
> 406 406 1val_4061 406
> 273 273 1val_2731 273
> 98NULLNULL98
> 213 213 1val_2131 213
> 255 NULLNULL255
> 401 401 1val_4011 401
> 278 NULLNULL278
> 6666 11val_6611 66
> 224 NULLNULL224
> 311 NULLNULL311
> {code}
>
> *ACTUAL RESULT*
> {code:java}
> 128 NULLNULL128
> 146 146 1val_1461 146
> 150 150 1val_1501 150
> 213 213 1val_2131 213
> 238 NULLNULL238
> 273 273 1val_2731 273
> 369 NULLNULL369
> 406 406 1val_4061 406
> 98NULLNULL98
> 401 401 1val_4011 401
> 6666 11val_6611 66
> {code}
>
> *ROOT CAUSE*
> src1 left join src2, src1 is big table and src2 is small table. Join result
> between big table row and the corresponding hashtable maybe NO_MATCH state,
> however, these NO_MATCH rows is needed because LEFT OUTER JOIN.
> In addition, these big table rows will not spilled into matchfile related to
> this hashtable on disk because only SPILL state can use `spillBigTableRow`.
> Then, these big table rows will be spilled into matchfile in hashtables of
> table `srcpart`(second small table)
> Finally, when reProcessBigTable, big table rows in matchfile are only read
> from `firstSmallTable`, some datum are missing.
>
> *WORKAROUND*
> configure firstSmallTable in completeInitializationOp and only spill big
> table row into firstSmallTable when spill matchfile.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[ https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naresh P R updated HIVE-23409: -- Attachment: HIVE-23409.1.patch > If TezSession application reopen fails for Timeline service down, default > TezSession from SessionPool is closed after a retry > - > > Key: HIVE-23409 > URL: https://issues.apache.org/jira/browse/HIVE-23409 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Major > Attachments: HIVE-23409.1.patch, HIVE-23409.patch > > > we are closing a default session from TezSessionPool at here. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589] > If all the sessions in a pool are destroyed, queries wait indefinitely at > TezSessionPool.getSession until HS2 restarts after other service recoveries. > [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:()) > - We are closing a default session because of retry failure. > It's better if we allow retry & fail than hung. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23447) Avoid sending configs to tasks and AM which are only relevant for HS2
[
https://issues.apache.org/jira/browse/HIVE-23447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105124#comment-17105124
]
Hive QA commented on HIVE-23447:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22278/dev-support/hive-personality.sh
|
| git revision | master / fdf6758 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22278/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22278/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid sending configs to tasks and AM which are only relevant for HS2
> -
>
> Key: HIVE-23447
> URL: https://issues.apache.org/jira/browse/HIVE-23447
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-23447.patch
>
>
> There are many configs which are only relevant for HS2. Longer term fix for
> this is to split HiveConf in multiple config classes relevant only for HS2,
> HMS, AM and tasks. And use only the objects in process where its relevant. In
> the interim, we can avoid configs with large value strings to send across.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23448) Remove hive-site.xml from input/output/processor payload
[ https://issues.apache.org/jira/browse/HIVE-23448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman reassigned HIVE-23448: --- > Remove hive-site.xml from input/output/processor payload > > > Key: HIVE-23448 > URL: https://issues.apache.org/jira/browse/HIVE-23448 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > > Depends on https://jira.apache.org/jira/browse/TEZ-4137?filter=-1 > We remove most xml configs from payloads in > https://jira.apache.org/jira/browse/HIVE-23175 > However, hive-site.xml could not be removed from those configs in early stage > for reasons outlined in that jira. > This Jira removes hive-site.xml configs from configuration just before > serializing payloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23344) Bump org.scala-lang:scala-compiler version to 2.12.4
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105228#comment-17105228 ] Peter Vary commented on HIVE-23344: --- +1 > Bump org.scala-lang:scala-compiler version to 2.12.4 > > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch, > HIVE-23344.02.patch, HIVE-23344.03.patch, HIVE-23344.03.patch, > HIVE-23344.04-dep-tree, HIVE-23344.04.patch > > > And bump up spark version, as 2.3.3 is not compatible with scala 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23450) TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky
[
https://issues.apache.org/jira/browse/HIVE-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105215#comment-17105215
]
Zoltan Haindrich commented on HIVE-23450:
-
there are a few different kind of error messages; I've attached a hive.log.gz
what I suspect right now is probably one of the following:
* druid is not yet ready - still starting up or something
* druid might be expecting a different version of calcite
> TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
> flaky
> --
>
> Key: HIVE-23450
> URL: https://issues.apache.org/jira/browse/HIVE-23450
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: hive.log.gz
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/22267/testReport/junit/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druid_materialized_view_rewrite_ssb_/
> error tends to be:
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 952.613 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [ERROR]
> testCliDriver[druid_materialized_view_rewrite_ssb](org.apache.hadoop.hive.cli.TestMiniDruidCliDriver)
> Time elapsed: 31.851 s <<< FAILURE!
> java.lang.AssertionError:
> Client execution failed with error code = 1
> running
> SELECT MONTH(`__time`) AS `mn___time_ok`,
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT) AS `qr___time_ok`,
> SUM(1) AS `sum_number_of_records_ok`,
> YEAR(`__time`) AS `yr___time_ok`
> FROM `ssb_mv_druid_100`
> GROUP BY MONTH(`__time`),
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT),
> YEAR(`__time`)
> fname=druid_materialized_view_rewrite_ssb.q
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23450) TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky
[
https://issues.apache.org/jira/browse/HIVE-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23450:
Attachment: hive.log.gz
> TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
> flaky
> --
>
> Key: HIVE-23450
> URL: https://issues.apache.org/jira/browse/HIVE-23450
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: hive.log.gz
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/22267/testReport/junit/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druid_materialized_view_rewrite_ssb_/
> error tends to be:
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 952.613 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [ERROR]
> testCliDriver[druid_materialized_view_rewrite_ssb](org.apache.hadoop.hive.cli.TestMiniDruidCliDriver)
> Time elapsed: 31.851 s <<< FAILURE!
> java.lang.AssertionError:
> Client execution failed with error code = 1
> running
> SELECT MONTH(`__time`) AS `mn___time_ok`,
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT) AS `qr___time_ok`,
> SUM(1) AS `sum_number_of_records_ok`,
> YEAR(`__time`) AS `yr___time_ok`
> FROM `ssb_mv_druid_100`
> GROUP BY MONTH(`__time`),
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT),
> YEAR(`__time`)
> fname=druid_materialized_view_rewrite_ssb.q
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23450) TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky
[
https://issues.apache.org/jira/browse/HIVE-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23450:
Parent: HIVE-22942
Issue Type: Sub-task (was: Bug)
> TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
> flaky
> --
>
> Key: HIVE-23450
> URL: https://issues.apache.org/jira/browse/HIVE-23450
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: hive.log.gz
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/22267/testReport/junit/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druid_materialized_view_rewrite_ssb_/
> error tends to be:
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 952.613 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [ERROR]
> testCliDriver[druid_materialized_view_rewrite_ssb](org.apache.hadoop.hive.cli.TestMiniDruidCliDriver)
> Time elapsed: 31.851 s <<< FAILURE!
> java.lang.AssertionError:
> Client execution failed with error code = 1
> running
> SELECT MONTH(`__time`) AS `mn___time_ok`,
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT) AS `qr___time_ok`,
> SUM(1) AS `sum_number_of_records_ok`,
> YEAR(`__time`) AS `yr___time_ok`
> FROM `ssb_mv_druid_100`
> GROUP BY MONTH(`__time`),
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT),
> YEAR(`__time`)
> fname=druid_materialized_view_rewrite_ssb.q
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23450) TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky
[
https://issues.apache.org/jira/browse/HIVE-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105216#comment-17105216
]
Zoltan Haindrich commented on HIVE-23450:
-
this test started falpping after the druid upgrade...before that I've not seen
it
cc: [~nishantbangarwa]
> TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
> flaky
> --
>
> Key: HIVE-23450
> URL: https://issues.apache.org/jira/browse/HIVE-23450
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: hive.log.gz
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/22267/testReport/junit/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druid_materialized_view_rewrite_ssb_/
> error tends to be:
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 952.613 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [ERROR]
> testCliDriver[druid_materialized_view_rewrite_ssb](org.apache.hadoop.hive.cli.TestMiniDruidCliDriver)
> Time elapsed: 31.851 s <<< FAILURE!
> java.lang.AssertionError:
> Client execution failed with error code = 1
> running
> SELECT MONTH(`__time`) AS `mn___time_ok`,
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT) AS `qr___time_ok`,
> SUM(1) AS `sum_number_of_records_ok`,
> YEAR(`__time`) AS `yr___time_ok`
> FROM `ssb_mv_druid_100`
> GROUP BY MONTH(`__time`),
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT),
> YEAR(`__time`)
> fname=druid_materialized_view_rewrite_ssb.q
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Attachment: HIVE-23432.01.patch Status: Patch Available (was: Open) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23437) Concurrent partition creation requests cause underlying HDFS folder to be deleted
[
https://issues.apache.org/jira/browse/HIVE-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105218#comment-17105218
]
Marc Demierre edited comment on HIVE-23437 at 5/12/20, 8:44 AM:
We tried a workaround on the client side to ensure the calls are not
simultaneous by delaying them. It didn't solve the issue, only made it rarer.
We also observed a second instance of the problem which is slightly different:
* T1:
** R1 creates the directory, then is paused/waiting
* T2:
** R2 arrives, does not create the directory as it exists
** R2 creates the partition (wins the race on DB) and completes
* T3:
** R1 resumes, sees that it failed the DB transaction, deletes the folder
Relevant logs (R1=2558, R2=2556):
{code:java}
2020-05-11 20:00:00,944 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:00,945 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2556: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,481 INFO [pool-7-thread-2558]: common.FileUtils
(FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,521 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(352)) - Updating partition stats
fast for: sbd_dev_debug_hive_4
2020-05-11 20:00:01,537 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(355)) - Updated size to 0
2020-05-11 20:00:01,764 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(41)) -
deleting
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]: fs.TrashPolicyDefault
(TrashPolicyDefault.java:moveToTrash(168)) - Moved:
'hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18'
to trash at:
hdfs://sbd/user/sbd-fh-connect-autodumphdfs/.Trash/Current/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(48)) - Moved
to trash:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,788 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(217)) -
Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error:
javax.jdo.JDODataStoreException: Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@3254e57d" using statement
"INSERT INTO "PARTITIONS"
("PART_ID","CREATE_TIME","LAST_ACCESS_TIME","PART_NAME","SD_ID","TBL_ID")
VALUES (?,?,?,?,?,?)" failed : ERROR: duplicate key value violates unique
constraint "UNIQUEPARTITION"
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,869 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
AlreadyExistsException(message:Partition already
exists:Partition(values:[ingestion, hourly, 2020-05-11, 18],
dbName:sbd_dev_autodump, tableName:sbd_dev_debug_hive_4, createTime:0,
lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name,
type:string, comment:null), FieldSchema(name:age, type:int, comment:null),
FieldSchema(name:salary, type:int,
[jira] [Commented] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[
https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105175#comment-17105175
]
Hive QA commented on HIVE-23409:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22279/dev-support/hive-personality.sh
|
| git revision | master / fdf6758 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22279/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> If TezSession application reopen fails for Timeline service down, default
> TezSession from SessionPool is closed after a retry
> -
>
> Key: HIVE-23409
> URL: https://issues.apache.org/jira/browse/HIVE-23409
> Project: Hive
> Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-23409.1.patch, HIVE-23409.patch
>
>
> we are closing a default session from TezSessionPool at here.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589]
> If all the sessions in a pool are destroyed, queries wait indefinitely at
> TezSessionPool.getSession until HS2 restarts after other service recoveries.
> [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:())
> - We are closing a default session because of retry failure.
> It's better if we allow retry & fail than hung.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23437) Concurrent partition creation requests cause underlying HDFS folder to be deleted
[
https://issues.apache.org/jira/browse/HIVE-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105218#comment-17105218
]
Marc Demierre edited comment on HIVE-23437 at 5/12/20, 8:25 AM:
After trying a workaround on the client side to ensure the calls are not
simultaneous by delaying them. It didn't solve the issue, only made it rarer.
We also observed a second instance of the problem:
* T1:
** R1 creates the directory, then is paused/waiting
* T2:
** R2 arrives, does not create the directory as it exists
** R2 creates the partition (wins the race on DB) and completes
* T3:
** R1 resumes, sees that it failed the DB transaction, deletes the folder
Relevant logs (R1=2558, R2=2556):
{code:java}
2020-05-11 20:00:00,944 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:00,945 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2556: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,481 INFO [pool-7-thread-2558]: common.FileUtils
(FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,521 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(352)) - Updating partition stats
fast for: sbd_dev_debug_hive_4
2020-05-11 20:00:01,537 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(355)) - Updated size to 0
2020-05-11 20:00:01,764 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(41)) -
deleting
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]: fs.TrashPolicyDefault
(TrashPolicyDefault.java:moveToTrash(168)) - Moved:
'hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18'
to trash at:
hdfs://sbd/user/sbd-fh-connect-autodumphdfs/.Trash/Current/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(48)) - Moved
to trash:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,788 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(217)) -
Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error:
javax.jdo.JDODataStoreException: Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@3254e57d" using statement
"INSERT INTO "PARTITIONS"
("PART_ID","CREATE_TIME","LAST_ACCESS_TIME","PART_NAME","SD_ID","TBL_ID")
VALUES (?,?,?,?,?,?)" failed : ERROR: duplicate key value violates unique
constraint "UNIQUEPARTITION"
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,869 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
AlreadyExistsException(message:Partition already
exists:Partition(values:[ingestion, hourly, 2020-05-11, 18],
dbName:sbd_dev_autodump, tableName:sbd_dev_debug_hive_4, createTime:0,
lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name,
type:string, comment:null), FieldSchema(name:age, type:int, comment:null),
FieldSchema(name:salary, type:int, comment:null)],
[jira] [Comment Edited] (HIVE-23437) Concurrent partition creation requests cause underlying HDFS folder to be deleted
[
https://issues.apache.org/jira/browse/HIVE-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105218#comment-17105218
]
Marc Demierre edited comment on HIVE-23437 at 5/12/20, 8:25 AM:
We tried a workaround on the client side to ensure the calls are not
simultaneous by delaying them. It didn't solve the issue, only made it rarer.
We also observed a second instance of the problem:
* T1:
** R1 creates the directory, then is paused/waiting
* T2:
** R2 arrives, does not create the directory as it exists
** R2 creates the partition (wins the race on DB) and completes
* T3:
** R1 resumes, sees that it failed the DB transaction, deletes the folder
Relevant logs (R1=2558, R2=2556):
{code:java}
2020-05-11 20:00:00,944 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:00,945 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2556: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,481 INFO [pool-7-thread-2558]: common.FileUtils
(FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,521 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(352)) - Updating partition stats
fast for: sbd_dev_debug_hive_4
2020-05-11 20:00:01,537 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(355)) - Updated size to 0
2020-05-11 20:00:01,764 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(41)) -
deleting
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]: fs.TrashPolicyDefault
(TrashPolicyDefault.java:moveToTrash(168)) - Moved:
'hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18'
to trash at:
hdfs://sbd/user/sbd-fh-connect-autodumphdfs/.Trash/Current/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(48)) - Moved
to trash:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,788 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(217)) -
Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error:
javax.jdo.JDODataStoreException: Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@3254e57d" using statement
"INSERT INTO "PARTITIONS"
("PART_ID","CREATE_TIME","LAST_ACCESS_TIME","PART_NAME","SD_ID","TBL_ID")
VALUES (?,?,?,?,?,?)" failed : ERROR: duplicate key value violates unique
constraint "UNIQUEPARTITION"
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,869 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
AlreadyExistsException(message:Partition already
exists:Partition(values:[ingestion, hourly, 2020-05-11, 18],
dbName:sbd_dev_autodump, tableName:sbd_dev_debug_hive_4, createTime:0,
lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name,
type:string, comment:null), FieldSchema(name:age, type:int, comment:null),
FieldSchema(name:salary, type:int, comment:null)],
[jira] [Commented] (HIVE-23437) Concurrent partition creation requests cause underlying HDFS folder to be deleted
[
https://issues.apache.org/jira/browse/HIVE-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105218#comment-17105218
]
Marc Demierre commented on HIVE-23437:
--
After trying a workaround on the client side to delay the calls, we observed a
second instance of the problem:
* T1:
** R1 creates the directory, then is paused/waiting
* T2:
** R2 arrives, does not create the directory as it exists
** R2 creates the partition (wins the race on DB) and completes
* T3:
** R1 resumes, sees that it failed the DB transaction, deletes the folder
Relevant logs (R1=2558, R2=2556):
{code:java}
2020-05-11 20:00:00,944 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:00,945 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2556: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,311 INFO [pool-7-thread-2556]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,481 INFO [pool-7-thread-2558]: common.FileUtils
(FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,521 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(352)) - Updating partition stats
fast for: sbd_dev_debug_hive_4
2020-05-11 20:00:01,537 WARN [pool-7-thread-2556]: hive.log
(MetaStoreUtils.java:updatePartitionStatsFast(355)) - Updated size to 0
2020-05-11 20:00:01,764 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(41)) -
deleting
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]: fs.TrashPolicyDefault
(TrashPolicyDefault.java:moveToTrash(168)) - Moved:
'hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18'
to trash at:
hdfs://sbd/user/sbd-fh-connect-autodumphdfs/.Trash/Current/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,787 INFO [pool-7-thread-2558]:
metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(48)) - Moved
to trash:
hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:01,788 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(217)) -
Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error:
javax.jdo.JDODataStoreException: Insert of object
"org.apache.hadoop.hive.metastore.model.MPartition@3254e57d" using statement
"INSERT INTO "PARTITIONS"
("PART_ID","CREATE_TIME","LAST_ACCESS_TIME","PART_NAME","SD_ID","TBL_ID")
VALUES (?,?,?,?,?,?)" failed : ERROR: duplicate key value violates unique
constraint "UNIQUEPARTITION"
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(775)) - 2558: append_partition_by_name:
db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,788 INFO [pool-7-thread-2558]: HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(319)) -
ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.2
cmd=append_partition_by_name: db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
part=time=ingestion/bucket=hourly/date=2020-05-11/hour=18
2020-05-11 20:00:03,869 ERROR [pool-7-thread-2558]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) -
AlreadyExistsException(message:Partition already
exists:Partition(values:[ingestion, hourly, 2020-05-11, 18],
dbName:sbd_dev_autodump, tableName:sbd_dev_debug_hive_4, createTime:0,
lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:name,
type:string, comment:null), FieldSchema(name:age, type:int, comment:null),
FieldSchema(name:salary, type:int, comment:null)],
location:hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=18,
[jira] [Updated] (HIVE-23448) Remove hive-site.xml from input/output/processor payload
[ https://issues.apache.org/jira/browse/HIVE-23448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman updated HIVE-23448: Attachment: HIVE-23448.patch > Remove hive-site.xml from input/output/processor payload > > > Key: HIVE-23448 > URL: https://issues.apache.org/jira/browse/HIVE-23448 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23448.patch > > > Depends on https://jira.apache.org/jira/browse/TEZ-4137?filter=-1 > We remove most xml configs from payloads in > https://jira.apache.org/jira/browse/HIVE-23175 > However, hive-site.xml could not be removed from those configs in early stage > for reasons outlined in that jira. > This Jira removes hive-site.xml configs from configuration just before > serializing payloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23447) Avoid sending configs to tasks and AM which are only relevant for HS2
[
https://issues.apache.org/jira/browse/HIVE-23447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105151#comment-17105151
]
Hive QA commented on HIVE-23447:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002666/HIVE-23447.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17265 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22278/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22278/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002666 - PreCommit-HIVE-Build
> Avoid sending configs to tasks and AM which are only relevant for HS2
> -
>
> Key: HIVE-23447
> URL: https://issues.apache.org/jira/browse/HIVE-23447
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-23447.patch
>
>
> There are many configs which are only relevant for HS2. Longer term fix for
> this is to split HiveConf in multiple config classes relevant only for HS2,
> HMS, AM and tasks. And use only the objects in process where its relevant. In
> the interim, we can avoid configs with large value strings to send across.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23450) TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is flaky
[
https://issues.apache.org/jira/browse/HIVE-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23450:
Summary:
TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
flaky (was:
TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] )
> TestMiniDruidCliDriver#testCliDriver[druid_materialized_view_rewrite_ssb] is
> flaky
> --
>
> Key: HIVE-23450
> URL: https://issues.apache.org/jira/browse/HIVE-23450
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/22267/testReport/junit/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druid_materialized_view_rewrite_ssb_/
> error tends to be:
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 952.613 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [ERROR]
> testCliDriver[druid_materialized_view_rewrite_ssb](org.apache.hadoop.hive.cli.TestMiniDruidCliDriver)
> Time elapsed: 31.851 s <<< FAILURE!
> java.lang.AssertionError:
> Client execution failed with error code = 1
> running
> SELECT MONTH(`__time`) AS `mn___time_ok`,
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT) AS `qr___time_ok`,
> SUM(1) AS `sum_number_of_records_ok`,
> YEAR(`__time`) AS `yr___time_ok`
> FROM `ssb_mv_druid_100`
> GROUP BY MONTH(`__time`),
> CAST((MONTH(`__time`) - 1) / 3 + 1 AS BIGINT),
> YEAR(`__time`)
> fname=druid_materialized_view_rewrite_ssb.q
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[
https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105966#comment-17105966
]
Hive QA commented on HIVE-23423:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
31s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 11 new + 580 unchanged - 3
fixed = 591 total (was 583) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22301/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22301/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22301/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22301/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hash aggregation is always disabled in vectorized execution of grouping set
> queries
> ---
>
> Key: HIVE-23423
> URL: https://issues.apache.org/jira/browse/HIVE-23423
> Project: Hive
> Issue Type: Bug
> Components: Hive, Operators, Query Processor
>Affects Versions: 4.0.0
>Reporter: Nita Dembla
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: Performance
> Attachments: HIVE-23423.1.patch, HIVE-23423.WIP.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with
> disabling hash aggregation on grouping set queries. Need a fix for
> VectorGroupbyOperator operator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[
https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105973#comment-17105973
]
De Li commented on HIVE-23452:
--
{code:java}
// code placeholder
2020-05-13 11:17:40,522 [ERROR] [InputInitializer {Map 1} #0]
|dao.GenericJdbcDatabaseAccessor|: Caught exception while trying to get the
number of records
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
'test_mysql.test_gp' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
at com.mysql.jdbc.Util.getInstance(Util.java:383)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3847)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3783)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2447)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2594)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1901)
at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2002)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getTotalNumberOfRecords(GenericJdbcDatabaseAccessor.java:122)
at
org.apache.hive.storage.jdbc.JdbcInputFormat.getSplits(JdbcInputFormat.java:73)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:765)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:244)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-13 11:17:40,522 [INFO] [InputInitializer {Map 2} #0] |exec.Utilities|:
PLAN PATH =
hdfs://ng4df3e47-master-instance-1pg5hd09.novalocal:8020/tmp/hive/hive/bb1e2741-8dac-4cc3-89b6-bad9c28d4b9f/hive_2020-05-13_11-17-33_149_996173199554056250-1/hive/_tez_scratch_dir/c04b135f-c4b2-4d00-a2dc-5308cc617e73/map.xml
2020-05-13 11:17:40,525 [ERROR] [InputInitializer {Map 1} #0]
|jdbc.JdbcInputFormat|: Error while splitting input data.
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
'test_mysql.test_gp' doesn't exist
at
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getTotalNumberOfRecords(GenericJdbcDatabaseAccessor.java:137)
at
org.apache.hive.storage.jdbc.JdbcInputFormat.getSplits(JdbcInputFormat.java:73)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:765)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:244)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[ https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105961#comment-17105961 ] Syed Shameerur Rahman commented on HIVE-23452: -- [~lide] Can you please share the stacktrace of exception? > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez > --- > > Key: HIVE-23452 > URL: https://issues.apache.org/jira/browse/HIVE-23452 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: De Li >Priority: Major > > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by > Tez and it works when query with MR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[
https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105941#comment-17105941
]
Rajesh Balamohan commented on HIVE-23451:
-
I had the same confusion on this codepath on the treatment for index "0" and .1
patch followed the approach of reducing duplicate invocation.
Looking at the codepath more reveals that {{totalFiles}} is set to 1 by default
in semanticAnalyzer. In case of bucketing, it ends up setting to number of
buckets.
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L6942]
(set to number of buckets)
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L6787]
(sets to 1 by default).
"0" index handling isn't needed actually. I will attach .2 version shortly.
> FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
> ---
>
> Key: HIVE-23451
> URL: https://issues.apache.org/jira/browse/HIVE-23451
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797]
> Can avoid a NN call here (i.e, mainly for small queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[
https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105975#comment-17105975
]
De Li commented on HIVE-23452:
--
{code:java}
// code placeholder
CREATE EXTERNAL TABLE IF NOT EXISTS test_gp_ex (
i int
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "POSTGRES",
"hive.sql.jdbc.driver" = "org.postgresql.Driver",
"hive.sql.jdbc.url" = "jdbc:postgresql://:5432/db",
"hive.sql.dbcp.username" = "gpadmin",
"hive.sql.dbcp.password" = "xxx",
"hive.sql.table" = "test_gp"
);
CREATE EXTERNAL TABLE IF NOT EXISTS test_mysql_table_db (
id int,
name String
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "MYSQL",
"hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
"hive.sql.jdbc.url" = "jdbc:mysql://xxx/test_mysql",
"hive.sql.dbcp.username" = "",
"hive.sql.dbcp.password" = "*",
"hive.sql.table" = "test_mysql.test_mysqltable"
);
{code}
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez
> ---
>
> Key: HIVE-23452
> URL: https://issues.apache.org/jira/browse/HIVE-23452
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.1.0
>Reporter: De Li
>Priority: Major
>
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by
> Tez and it works when query with MR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[
https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105984#comment-17105984
]
Syed Shameerur Rahman commented on HIVE-23452:
--
[~lide] It seems the table doesn't exists
{code:java}
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
'test_mysql.test_gp' doesn't exist
{code}
Try with just *test_gp*
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez
> ---
>
> Key: HIVE-23452
> URL: https://issues.apache.org/jira/browse/HIVE-23452
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.1.0
>Reporter: De Li
>Priority: Major
>
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by
> Tez and it works when query with MR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23451: Attachment: HIVE-23451.2.patch > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23449: Attachment: HIVE-23449.3.patch > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, > HIVE-23449.3.patch, Screenshot 2020-05-12 at 1.09.35 PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[ https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105931#comment-17105931 ] Ashutosh Chauhan commented on HIVE-23409: - +1 > If TezSession application reopen fails for Timeline service down, default > TezSession from SessionPool is closed after a retry > - > > Key: HIVE-23409 > URL: https://issues.apache.org/jira/browse/HIVE-23409 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Major > Attachments: HIVE-23409.1.patch, HIVE-23409.patch > > > we are closing a default session from TezSessionPool at here. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589] > If all the sessions in a pool are destroyed, queries wait indefinitely at > TezSessionPool.getSession until HS2 restarts after other service recoveries. > [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:()) > - We are closing a default session because of retry failure. > It's better if we allow retry & fail than hung. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
[ https://issues.apache.org/jira/browse/HIVE-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23409: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Naresh! > If TezSession application reopen fails for Timeline service down, default > TezSession from SessionPool is closed after a retry > - > > Key: HIVE-23409 > URL: https://issues.apache.org/jira/browse/HIVE-23409 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23409.1.patch, HIVE-23409.patch > > > we are closing a default session from TezSessionPool at here. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589] > If all the sessions in a pool are destroyed, queries wait indefinitely at > TezSessionPool.getSession until HS2 restarts after other service recoveries. > [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:()) > - We are closing a default session because of retry failure. > It's better if we allow retry & fail than hung. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105932#comment-17105932 ] Rajesh Balamohan commented on HIVE-23449: - .3 is same as .2 with minor comment added. Thanks for the review [~ashutoshc] . > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, > HIVE-23449.3.patch, Screenshot 2020-05-12 at 1.09.35 PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105948#comment-17105948
]
Hive QA commented on HIVE-23354:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 2 new + 108 unchanged - 2
fixed = 110 total (was 110) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22300/dev-support/hive-personality.sh
|
| git revision | master / 2ff6370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22300/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22300/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch
>
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due to various factors such as file
> compression and the order in which values are written. It may be prudent to
> trust
[jira] [Commented] (HIVE-23455) Improve error message for external orc table
[
https://issues.apache.org/jira/browse/HIVE-23455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105967#comment-17105967
]
Mustafa Iman commented on HIVE-23455:
-
[~ashutoshc] The error happens when populating loop does not visit all the
indices. The loop does not visit all the indices -> there are nulls in the
array. Therefore the extra assertion loop. Regarding TestNegativeCliDriver, I
tried that but it shows only generic task error at the end of the output in
both cases.
Anyway, [~gopalv] thinks this is breaking schema evolution. I still do not see
it as this codepath fails in very next method
call(EncodedTreeReaderFactory#createRootTreeReader) with NullPointerException
anyway.
I also tried the sample query he gave with mini llap drivers:
{code:java}
create external table foo(x int);
insert into foo values(1);
alter table foo add column (y int);
select sum(x) from foo;
{code}
It works for me. So I did not close the ticket but please do not merge yet.
> Improve error message for external orc table
>
>
> Key: HIVE-23455
> URL: https://issues.apache.org/jira/browse/HIVE-23455
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23455.patch, acid_table_as_external.q,
> acid_table_as_external.q.out
>
>
> Since there is no schema validation for external tables, users may face
> various errors if their orc data and external table schema does not match. If
> orc schema has fewer columns than projection OrcEncodedDataConsumer may
> receive an incomplete TypeDescription array which will manifest itself as
> NullPointerException later.
> We can at least verify that OrcEncodedDataConsumer gets enough
> TypeDescriptions. If assertion fails, user sees there is something wrong with
> the schema and hopefully resolves the problem quickly. If there are enough
> columns in the file but the schema of the query does not match, user
> generally sees a ClassCastException. If there are enough columns and types
> accidentally match, there is nothing we can do as this is an external table.
> We have seen this when trying to use a managed table as external table
> location. Although user facing schemas are the same, managed table has acid
> related metadata. I am adding a q file demonstrating NullPointerException
> with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added
> this to precommit tests as it is hard to assert the exception message from
> mini driver framework and effectively it is just changing the error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries
[
https://issues.apache.org/jira/browse/HIVE-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105983#comment-17105983
]
Hive QA commented on HIVE-23423:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002774/HIVE-23423.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17269 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22301/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22301/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22301/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002774 - PreCommit-HIVE-Build
> Hash aggregation is always disabled in vectorized execution of grouping set
> queries
> ---
>
> Key: HIVE-23423
> URL: https://issues.apache.org/jira/browse/HIVE-23423
> Project: Hive
> Issue Type: Bug
> Components: Hive, Operators, Query Processor
>Affects Versions: 4.0.0
>Reporter: Nita Dembla
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: Performance
> Attachments: HIVE-23423.1.patch, HIVE-23423.WIP.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with
> disabling hash aggregation on grouping set queries. Need a fix for
> VectorGroupbyOperator operator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23446) LLAP: Reduce IPC connection misses to AM for short queries
[
https://issues.apache.org/jira/browse/HIVE-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105362#comment-17105362
]
Rajesh Balamohan commented on HIVE-23446:
-
RB link: https://reviews.apache.org/r/72499/diff/1#index_header
> LLAP: Reduce IPC connection misses to AM for short queries
> --
>
> Key: HIVE-23446
> URL: https://issues.apache.org/jira/browse/HIVE-23446
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23446.1.patch
>
>
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java#L343]
>
> Umbilical UGI pool for is maintained at QueryInfo level. When there are lots
> of short queries, this ends up missing IPC cache and ends up recreating
> threads/connections to the same AM.
> It would be good to maintain this pool in {{ContainerRunnerImpl}} instead and
> recycle as needed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23053) Clean Up Stats Mergers
[ https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23053: -- Attachment: HIVE-23053.3.patch > Clean Up Stats Mergers > -- > > Key: HIVE-23053 > URL: https://issues.apache.org/jira/browse/HIVE-23053 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23053.1.patch, HIVE-23053.2.patch, > HIVE-23053.2.patch, HIVE-23053.3.patch, HIVE-23053.3.patch > > > * DEBUG log each invocation of merge > * Use the inherited logger properly > * Use parameterized logging > * Simplify code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23445) Remove mapreduce.workflow.* configs
[ https://issues.apache.org/jira/browse/HIVE-23445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23445: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. > Remove mapreduce.workflow.* configs > --- > > Key: HIVE-23445 > URL: https://issues.apache.org/jira/browse/HIVE-23445 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23445.patch > > > These configs were introduced in HIVE-3708 in the hope to develop tools to > visualize and monitor multiple MR jobs from Hive back in a day when MR was > used. Even that time in spite of these config additions, no such tools were > developed AFAIK. And now MR is hardly ever used. We can get rid of these > configs. That will help to reduce the size of HiveConf by a bit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[
https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105450#comment-17105450
]
Hive QA commented on HIVE-23451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002702/HIVE-23451.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17266 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=31)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=32)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22284/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002702 - PreCommit-HIVE-Build
> FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
> ---
>
> Key: HIVE-23451
> URL: https://issues.apache.org/jira/browse/HIVE-23451
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23451.1.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797]
> Can avoid a NN call here (i.e, mainly for small queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[
https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105456#comment-17105456
]
Hive QA commented on HIVE-22869:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002707/HIVE-22869.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22285/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-12 13:56:04.911
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22285/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-12 13:56:04.913
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
7ebc546..57c1593 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 7ebc546 HIVE-23442: ACID major compaction doesn't read base
directory correctly if it was written by insert overwrite (Marta Kuczora,
reviewed by Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 57c1593 HIVE-23445 : Remove mapreduce.workflow.* configs
(Ashutosh Chauhan via Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-12 13:56:06.239
+ rm -rf ../yetus_PreCommit-HIVE-Build-22285
+ mkdir ../yetus_PreCommit-HIVE-Build-22285
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22285
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22285/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java:
does not exist in index
error:
a/standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java:
does not exist in index
error:
a/standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java:
does not exist in index
error:
a/standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java:
does not exist in index
error:
a/standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java:
does not exist in index
Trying to apply the patch with -p1
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc7980336799039884338.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc7980336799039884338.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing:
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[
https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105369#comment-17105369
]
Hive QA commented on HIVE-23133:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 2 unchanged - 0 fixed
= 4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22283/dev-support/hive-personality.sh
|
| git revision | master / 8bfdd18 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22283/yetus/diff-checkstyle-ql.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22283/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Numeric operations can have different result across hardware archs
> --
>
> Key: HIVE-23133
> URL: https://issues.apache.org/jira/browse/HIVE-23133
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zhenyu Zheng
>Assignee: Zhenyu Zheng
>Priority: Major
> Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch,
> HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch
>
>
> Currently, we have set up an ARM CI to test out how Hive works on ARM
> platform:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
> Among the failures, we have observed that some numeric operations can have
> different result across hardware archs, such as:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
>
[jira] [Assigned] (HIVE-23446) LLAP: Reduce IPC connection misses to AM for short queries
[
https://issues.apache.org/jira/browse/HIVE-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-23446:
---
Assignee: László Bodor (was: Rajesh Balamohan)
> LLAP: Reduce IPC connection misses to AM for short queries
> --
>
> Key: HIVE-23446
> URL: https://issues.apache.org/jira/browse/HIVE-23446
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23446.1.patch
>
>
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java#L343]
>
> Umbilical UGI pool for is maintained at QueryInfo level. When there are lots
> of short queries, this ends up missing IPC cache and ends up recreating
> threads/connections to the same AM.
> It would be good to maintain this pool in {{ContainerRunnerImpl}} instead and
> recycle as needed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23446) LLAP: Reduce IPC connection misses to AM for short queries
[
https://issues.apache.org/jira/browse/HIVE-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-23446:
---
Assignee: Rajesh Balamohan (was: László Bodor)
> LLAP: Reduce IPC connection misses to AM for short queries
> --
>
> Key: HIVE-23446
> URL: https://issues.apache.org/jira/browse/HIVE-23446
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23446.1.patch
>
>
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java#L343]
>
> Umbilical UGI pool for is maintained at QueryInfo level. When there are lots
> of short queries, this ends up missing IPC cache and ends up recreating
> threads/connections to the same AM.
> It would be good to maintain this pool in {{ContainerRunnerImpl}} instead and
> recycle as needed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[
https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105393#comment-17105393
]
Hive QA commented on HIVE-23133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002694/HIVE-23133.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=54)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorTableWithBase
(batchId=256)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=215)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22283/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22283/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22283/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002694 - PreCommit-HIVE-Build
> Numeric operations can have different result across hardware archs
> --
>
> Key: HIVE-23133
> URL: https://issues.apache.org/jira/browse/HIVE-23133
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zhenyu Zheng
>Assignee: Zhenyu Zheng
>Priority: Major
> Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch,
> HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch
>
>
> Currently, we have set up an ARM CI to test out how Hive works on ARM
> platform:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
> Among the failures, we have observed that some numeric operations can have
> different result across hardware archs, such as:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/]
> we can see that the calculation results of log, exp, cos, toRadians etc is
> slitly different than the .out file results that we are
> comparing(they are tested and wrote on X86 machines), this is because of we
> use [Math
> Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for
> these kind of calculations.
> and according to the
> [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]:
> _Unlike some of the numeric methods of class StrictMath, all implementations
> of the equivalent functions of class Math are not_
> _defined to return the bit-for-bit same results. This relaxation permits
> better-performing implementations where strict reproducibility_
> _is not required._
> _By default many of the Math methods simply call the equivalent method in
> StrictMath for their implementation._
> _Code generators are encouraged to use platform-specific native libraries or
> microprocessor instructions, where available,_
> _to provide higher-performance implementations of Math methods._
> so the result will have difference across hardware archs.
> On the other hand, JAVA provided another library
> [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]
> that will not have this kind of problem as according to its'
> [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]:
> _To help ensure portability of Java programs, the definitions of some of the
> numeric functions in this package require that they produce_
> _the same results as certain published algorithms._
> So in order to fix the above mentioned problem, we have to consider switch to
> use StrictMath instead of Math.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23017) Use Inherited Logger in Tasks
[
https://issues.apache.org/jira/browse/HIVE-23017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23017:
--
Attachment: (was: HIVE-23053.3.patch)
> Use Inherited Logger in Tasks
> -
>
> Key: HIVE-23017
> URL: https://issues.apache.org/jira/browse/HIVE-23017
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23017.1.patch, HIVE-23017.2.patch
>
>
> {code:java|title=FetchTask}
> public class FetchTask extends Task implements Serializable {
> private static final long serialVersionUID = 1L;
> private int maxRows = 100;
> private FetchOperator fetch;
> private ListSinkOperator sink;
> private int totalRows;
> private static transient final Logger LOG =
> LoggerFactory.getLogger(FetchTask.class);
> JobConf job = null;
> {code}
> The Tasks class is base class for many other. Cleaner to simply make the
> {{Logger}} {{protected}} and usable by other classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23017) Use Inherited Logger in Tasks
[
https://issues.apache.org/jira/browse/HIVE-23017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23017:
--
Attachment: HIVE-23053.3.patch
> Use Inherited Logger in Tasks
> -
>
> Key: HIVE-23017
> URL: https://issues.apache.org/jira/browse/HIVE-23017
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23017.1.patch, HIVE-23017.2.patch
>
>
> {code:java|title=FetchTask}
> public class FetchTask extends Task implements Serializable {
> private static final long serialVersionUID = 1L;
> private int maxRows = 100;
> private FetchOperator fetch;
> private ListSinkOperator sink;
> private int totalRows;
> private static transient final Logger LOG =
> LoggerFactory.getLogger(FetchTask.class);
> JobConf job = null;
> {code}
> The Tasks class is base class for many other. Cleaner to simply make the
> {{Logger}} {{protected}} and usable by other classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23442) ACID major compaction doesn't read base directory correctly if it was written by insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105338#comment-17105338
]
Hive QA commented on HIVE-23442:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002691/HIVE-23442.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17266 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002691 - PreCommit-HIVE-Build
> ACID major compaction doesn't read base directory correctly if it was written
> by insert overwrite
> -
>
> Key: HIVE-23442
> URL: https://issues.apache.org/jira/browse/HIVE-23442
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23442.1.patch, HIVE-23442.2.patch
>
>
> Steps to reproduce:
> {noformat}
> SET hive.acid.direct.insert.enabled=true;
> CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
> INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
> CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC
> TBLPROPERTIES('transactional'='true');
> INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
> UPDATE test_comp SET b=55, c=66 WHERE a=2;
> DELETE FROM test_comp WHERE a=4;
> UPDATE test_comp SET b=77 WHERE a=1;
> SELECT * FROM test_comp;
> 3 3 3
> 2 55 66
> 1 77 1
> ALTER TABLE test_comp COMPACT 'MAJOR';
> SELECT * FROM test_comp;
> 2 55 66
> 1 77 1
> {noformat}
> This issue only occurs if the base directory was created with an insert
> overwrite command and the hive.acid.direct.insert.enabled parameter was true.
> This issue doesn't affect the query based compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22869) Add locking benchmark to metastore-tools/metastore-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-22869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22869: - Attachment: HIVE-22869.5.patch > Add locking benchmark to metastore-tools/metastore-benchmarks > - > > Key: HIVE-22869 > URL: https://issues.apache.org/jira/browse/HIVE-22869 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-22869.2.patch, HIVE-22869.3.patch, > HIVE-22869.4.patch, HIVE-22869.5.patch, HIVE-22869.patch > > > Add the possibility to run benchmarks on opening lock in the HMS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23442) ACID major compaction doesn't read base directory correctly if it was written by insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-23442:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ACID major compaction doesn't read base directory correctly if it was written
> by insert overwrite
> -
>
> Key: HIVE-23442
> URL: https://issues.apache.org/jira/browse/HIVE-23442
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23442.1.patch, HIVE-23442.2.patch
>
>
> Steps to reproduce:
> {noformat}
> SET hive.acid.direct.insert.enabled=true;
> CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
> INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
> CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC
> TBLPROPERTIES('transactional'='true');
> INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
> UPDATE test_comp SET b=55, c=66 WHERE a=2;
> DELETE FROM test_comp WHERE a=4;
> UPDATE test_comp SET b=77 WHERE a=1;
> SELECT * FROM test_comp;
> 3 3 3
> 2 55 66
> 1 77 1
> ALTER TABLE test_comp COMPACT 'MAJOR';
> SELECT * FROM test_comp;
> 2 55 66
> 1 77 1
> {noformat}
> This issue only occurs if the base directory was created with an insert
> overwrite command and the hive.acid.direct.insert.enabled parameter was true.
> This issue doesn't affect the query based compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23442) ACID major compaction doesn't read base directory correctly if it was written by insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105351#comment-17105351
]
Marta Kuczora commented on HIVE-23442:
--
Pushed to master.
Thanks a lot [~pvary] for the review!
> ACID major compaction doesn't read base directory correctly if it was written
> by insert overwrite
> -
>
> Key: HIVE-23442
> URL: https://issues.apache.org/jira/browse/HIVE-23442
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23442.1.patch, HIVE-23442.2.patch
>
>
> Steps to reproduce:
> {noformat}
> SET hive.acid.direct.insert.enabled=true;
> CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
> INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
> CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC
> TBLPROPERTIES('transactional'='true');
> INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
> UPDATE test_comp SET b=55, c=66 WHERE a=2;
> DELETE FROM test_comp WHERE a=4;
> UPDATE test_comp SET b=77 WHERE a=1;
> SELECT * FROM test_comp;
> 3 3 3
> 2 55 66
> 1 77 1
> ALTER TABLE test_comp COMPACT 'MAJOR';
> SELECT * FROM test_comp;
> 2 55 66
> 1 77 1
> {noformat}
> This issue only occurs if the base directory was created with an insert
> overwrite command and the hive.acid.direct.insert.enabled parameter was true.
> This issue doesn't affect the query based compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[
https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105419#comment-17105419
]
Hive QA commented on HIVE-23451:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22284/dev-support/hive-personality.sh
|
| git revision | master / 7ebc546 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22284/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
> ---
>
> Key: HIVE-23451
> URL: https://issues.apache.org/jira/browse/HIVE-23451
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23451.1.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797]
> Can avoid a NN call here (i.e, mainly for small queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23099:
--
Attachment: HIVE-23099.4.patch
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch,
> HIVE-23099.4.patch, HIVE-23099.4.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes and repl dump
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23433 started by Aasha Medhi. -- > Add Deny Policy on Target Database After Ranger Replication to avoid writes > and repl dump > - > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23442) ACID major compaction doesn't read base directory correctly if it was written by insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-23442:
-
Attachment: HIVE-23442.2.patch
> ACID major compaction doesn't read base directory correctly if it was written
> by insert overwrite
> -
>
> Key: HIVE-23442
> URL: https://issues.apache.org/jira/browse/HIVE-23442
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23442.1.patch, HIVE-23442.2.patch
>
>
> Steps to reproduce:
> {noformat}
> SET hive.acid.direct.insert.enabled=true;
> CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
> INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
> CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC
> TBLPROPERTIES('transactional'='true');
> INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
> UPDATE test_comp SET b=55, c=66 WHERE a=2;
> DELETE FROM test_comp WHERE a=4;
> UPDATE test_comp SET b=77 WHERE a=1;
> SELECT * FROM test_comp;
> 3 3 3
> 2 55 66
> 1 77 1
> ALTER TABLE test_comp COMPACT 'MAJOR';
> SELECT * FROM test_comp;
> 2 55 66
> 1 77 1
> {noformat}
> This issue only occurs if the base directory was created with an insert
> overwrite command and the hive.acid.direct.insert.enabled parameter was true.
> This issue doesn't affect the query based compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105253#comment-17105253
]
Hive QA commented on HIVE-23432:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 29 new + 7 unchanged - 2 fixed
= 36 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
53s{color} | {color:red} ql generated 16 new + 1527 unchanged - 0 fixed = 1543
total (was 1527) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of rangerExportPolicyList at line 95 of value previously
dereferenced in org.apache.hadoop.hive.ql.exec.repl.RangerLoadTask.execute()
At RangerLoadTask.java:95 of value previously dereferenced in
org.apache.hadoop.hive.ql.exec.repl.RangerLoadTask.execute() At
RangerLoadTask.java:[line 95] |
| | Unread field:RangerDumpBegin.java:[line 36] |
| | Unread field:RangerDumpBegin.java:[line 37] |
| | Unread field:RangerDumpEnd.java:[line 45] |
| | Unread field:RangerDumpEnd.java:[line 44] |
| | Unread field:RangerDumpEnd.java:[line 47] |
| | Unread field:RangerDumpEnd.java:[line 46] |
| | Unread field:RangerLoadBegin.java:[line 44] |
| | Unread field:RangerLoadBegin.java:[line 45] |
| | Unread field:RangerLoadBegin.java:[line 42] |
| | Unread field:RangerLoadBegin.java:[line 43] |
| | Unread field:RangerLoadEnd.java:[line 50] |
| | Unread field:RangerLoadEnd.java:[line 52] |
| | Unread field:RangerLoadEnd.java:[line 51] |
| | Unread field:RangerLoadEnd.java:[line 48] |
| | Unread field:RangerLoadEnd.java:[line 49] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22280/dev-support/hive-personality.sh
|
| git revision | master / fdf6758 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22280/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22280/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23451: Attachment: HIVE-23451.1.patch > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23451: Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23437) Concurrent partition creation requests cause underlying HDFS folder to be deleted
[
https://issues.apache.org/jira/browse/HIVE-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marc Demierre updated HIVE-23437:
-
Environment:
* Hive version: 1.2.1000.2.6.5.0-292 (HDP 2.6.5)
* HDFS version: 2.7.3.2.6.5.0-292 (HDP 2.6.5)
* Metastore DB: PostgreSQL 9.4
was:
* Hive version: 1.2.1000.2.6.5.0-292 (HDP 2.6.5)
* HDFS version: 2.7.3.2.6.5.0-292 (HDP 2.6.5)
> Concurrent partition creation requests cause underlying HDFS folder to be
> deleted
> -
>
> Key: HIVE-23437
> URL: https://issues.apache.org/jira/browse/HIVE-23437
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 2.1.0, 3.1.2
> Environment: * Hive version: 1.2.1000.2.6.5.0-292 (HDP 2.6.5)
> * HDFS version: 2.7.3.2.6.5.0-292 (HDP 2.6.5)
> * Metastore DB: PostgreSQL 9.4
>Reporter: Marc Demierre
>Priority: Major
>
> There seems to be a race condition in Hive Metasore when issuing several
> concurrent partition creation requests for the same new partition.
> In our case, this triggered due to Kafka Connect Hive integration, which
> fires simultaneous partition creation requests from all its tasks when
> syncing to Hive.
> We are running HDP 2.6.5 but a quick survey of the upstream code still shows
> the same in 3.1.2 (latest Hive release).
> Our investigation pointed to the following code (here in Hive 2.1.0, the base
> for HDP 2.6.5):
> [https://github.com/apache/hive/blob/rel/release-2.1.0/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L2127]
> Same code in 3.1.2:
> https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3202
> The generic scenario is the following:
> # T1 (time period 1):
> ** R1 (request 1) creates the HDFS dir
> ** R2 also tries creating the HDFS dir
> ** Both succeed (as if it already exists it succeeds, R1/R2 could be
> interverted)
> # T2:
> ** R1 creates the partition in metastore DB, all OK
> # T3:
> ** R2 tries to create partition in metastore DB, gets exception from DB
> because it exists. Rollback transaction.
> ** R2 thinks it created the directory (in fact they both did we do not know
> who), so it removes it
> # T4: State is invalid:
> ## Partition exists
> ## HDFS folder does not exist
> ## Some Hive/Spark queries fail when trying to use the folder
> Here are some logs of the issue happening on our cluster in a standalone
> metastore (R1 = thread 2303, R2 = thread 2302):
> {code:java}
> 2020-05-11 13:43:46,379 INFO [pool-7-thread-2303]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(775)) - 2303: append_partition_by_name:
> db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4 part=time=ingestion/buc
> ket=hourly/date=2020-05-11/hour=11
> 2020-05-11 13:43:46,379 INFO [pool-7-thread-2302]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(775)) - 2302: append_partition_by_name:
> db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4 part=time=ingestion/buc
> ket=hourly/date=2020-05-11/hour=11
> 2020-05-11 13:43:46,379 INFO [pool-7-thread-2303]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(319)) -
> ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.1
> cmd=append_partition_by_name
> : db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
> part=time=ingestion/bucket=hourly/date=2020-05-11/hour=11
> 2020-05-11 13:43:46,379 INFO [pool-7-thread-2302]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(319)) -
> ugi=sbd-fh-connect-autodumph...@sbd.corproot.net ip=10.222.76.1
> cmd=append_partition_by_name
> : db=sbd_dev_autodump tbl=sbd_dev_debug_hive_4
> part=time=ingestion/bucket=hourly/date=2020-05-11/hour=11
> 2020-05-11 13:43:47,953 INFO [pool-7-thread-2302]: common.FileUtils
> (FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
> hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly
> /date=2020-05-11/hour=11
> 2020-05-11 13:43:47,957 INFO [pool-7-thread-2303]: common.FileUtils
> (FileUtils.java:mkdir(573)) - Creating directory if it doesn't exist:
> hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly
> /date=2020-05-11/hour=11
> 2020-05-11 13:43:47,986 INFO [pool-7-thread-2302]:
> metastore.hivemetastoressimpl (HiveMetaStoreFsImpl.java:deleteDir(41)) -
> deleting
> hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/dat
> e=2020-05-11/hour=11
> 2020-05-11 13:43:47,992 INFO [pool-7-thread-2302]: fs.TrashPolicyDefault
> (TrashPolicyDefault.java:moveToTrash(168)) - Moved:
> 'hdfs://sbd/data/sbd/dev/sbd.dev.debug-hive-4/time=ingestion/bucket=hourly/date=2020-05-11/hour=11'
> to trash at:
>
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105283#comment-17105283
]
Hive QA commented on HIVE-23432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002688/HIVE-23432.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17267 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testRangerReplication
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testRangerReplication
(batchId=195)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002688 - PreCommit-HIVE-Build
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23442) ACID major compaction doesn't read base directory correctly if it was written by insert overwrite
[
https://issues.apache.org/jira/browse/HIVE-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105303#comment-17105303
]
Peter Vary commented on HIVE-23442:
---
+1 pending tests
> ACID major compaction doesn't read base directory correctly if it was written
> by insert overwrite
> -
>
> Key: HIVE-23442
> URL: https://issues.apache.org/jira/browse/HIVE-23442
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23442.1.patch, HIVE-23442.2.patch
>
>
> Steps to reproduce:
> {noformat}
> SET hive.acid.direct.insert.enabled=true;
> CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
> INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);
> CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC
> TBLPROPERTIES('transactional'='true');
> INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;
> UPDATE test_comp SET b=55, c=66 WHERE a=2;
> DELETE FROM test_comp WHERE a=4;
> UPDATE test_comp SET b=77 WHERE a=1;
> SELECT * FROM test_comp;
> 3 3 3
> 2 55 66
> 1 77 1
> ALTER TABLE test_comp COMPACT 'MAJOR';
> SELECT * FROM test_comp;
> 2 55 66
> 1 77 1
> {noformat}
> This issue only occurs if the base directory was created with an insert
> overwrite command and the hive.acid.direct.insert.enabled parameter was true.
> This issue doesn't affect the query based compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23436) Staging directory is not removed for stats gathering tasks
[ https://issues.apache.org/jira/browse/HIVE-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23436: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~kgyrtkirk] for the review! > Staging directory is not removed for stats gathering tasks > -- > > Key: HIVE-23436 > URL: https://issues.apache.org/jira/browse/HIVE-23436 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23436.02.patch, HIVE-23436.patch > > > When running a query which generates stats, then the staging directory is not > removed when the query is finished -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23449: Attachment: HIVE-23449.1.patch > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23449: Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Attachment: HIVE-23133.5.patch Status: Patch Available (was: Open) > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Status: Open (was: Patch Available) > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105308#comment-17105308
]
Panagiotis Garefalakis commented on HIVE-23443:
---
Hey [~prasanth_j] the described logic does make sense to me – we should
probably write it down with capital letters somewhere in TaskExecutorService
class :)
Anyway, left some comments in the PR let me know what you think
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23443.1.patch
>
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23438) Missing Rows When Left Outer Join In N-way HybridGraceHashJoin
[
https://issues.apache.org/jira/browse/HIVE-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105304#comment-17105304
]
Hive QA commented on HIVE-23438:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002689/HIVE-23438.branch-2.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 10635 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_union_src]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union24] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_fast_stats]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_1]
(batchId=87)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union24]
(batchId=123)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=264)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testNonAcidToAcidConversion02
(batchId=276)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=273)
org.apache.hive.beeline.cli.TestHiveCli.testCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testDatabaseOptions (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testErrOutput (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testHelp (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testInValidCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidDatabaseOptions (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testInvalidOptions2 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSetPromptValue (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd2 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSourceCmd3 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmd (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testSqlFromCmdWithDBName (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB1 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB2 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testUseCurrentDB3 (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testUseInvalidDB (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariables (batchId=173)
org.apache.hive.beeline.cli.TestHiveCli.testVariablesForSource (batchId=173)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22281/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22281/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22281/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 44 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002689 -
