[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: In Progress (was: Patch Available) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, HIVE-23340.6.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: HIVE-23340.7.patch > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: Patch Available (was: In Progress) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: In Progress (was: Patch Available) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Status: Patch Available (was: In Progress) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23550) GetSplits does not retries queries for CacliteSemanticException
[
https://issues.apache.org/jira/browse/HIVE-23550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao resolved HIVE-23550.
Fix Version/s: (was: 4.0.0)
Resolution: Invalid
HIVE-21641 changed the usage of CalciteAnalyzer from just
CalciteAnalyzer.genLogicalPlan to fully analyzing (CalciteAnalyzer.analyze) the
query. That includes the retry logic for calcite semantic exceptions too.
Closing this.
> GetSplits does not retries queries for CacliteSemanticException
> ---
>
> Key: HIVE-23550
> URL: https://issues.apache.org/jira/browse/HIVE-23550
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 3.1.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Major
>
> Reproducible case:
> {noformat}
> create table t1 (c1 int, c2 int, c3 int);
> select get_splits("select c2, count(distinct c3) from t1 group by c2 having
> count(distinct c3) > 1",0);{noformat}
>
> Error:
> {noformat}
> Error: java.io.IOException:
> org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException:
> Distinct without an aggregation. (state=,code=0)
> {noformat}
> This happens because calcite does not understand the query "select c2,
> count(distinct c3) from t1 group by c2 having count(distinct c3) > 1" and
> throws calciteSemanticException
>
> If this query is run directly via beeline, hiveserver2 catches this exception
> and re-analyzes the query by turning off the cbo.
>
> This retrying mechanism is missing in GetSplits UDF.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23462) Add option to rewrite CUME_DIST to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?focusedWorklogId=440740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440740
]
ASF GitHub Bot logged work on HIVE-23462:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 12:20
Start Date: 03/Jun/20 12:20
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1031:
URL: https://github.com/apache/hive/pull/1031#discussion_r434513609
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java
##
@@ -68,25 +82,32 @@
* ⇒ SELECT ds_kll_quantile(ds_kll_sketch(CAST(id AS FLOAT)), 0.2) FROM
sketch_input;
*
*
+ * {@code cume_dist() over (order by id)}
Review comment:
I think these apidoc could be moved to the rewrite-rules - but they also
have there meaning here as well...maybe move them and add a more brief
description here?
##
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##
@@ -2495,19 +2495,22 @@ private static void
populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal
HIVE_OPTIMIZE_BI_REWRITE_COUNTDISTINCT_ENABLED("hive.optimize.bi.rewrite.countdistinct.enabled",
true,
"Enables to rewrite COUNT(DISTINCT(X)) queries to be rewritten to use
sketch functions."),
-HIVE_OPTIMIZE_BI_REWRITE_COUNT_DISTINCT_SKETCH(
-"hive.optimize.bi.rewrite.countdistinct.sketch", "hll",
+
HIVE_OPTIMIZE_BI_REWRITE_COUNT_DISTINCT_SKETCH("hive.optimize.bi.rewrite.countdistinct.sketch",
"hll",
new StringSet("hll"),
Review comment:
about enabling other sketches for count-distinct: I think they should
just work - however they might need a little testing; probably more important
would be to provide some way to change sketch construction
parameters...actually for our rewrites the sketch type could be considered as
part of the parameters
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
##
@@ -165,4 +166,10 @@ protected boolean shouldMergeProject() {
return false;
}
+ /** Make the method visible */
+ @Override
+ public AggCall aggregateCall(SqlAggFunction aggFunction, boolean distinct,
boolean approximate, boolean ignoreNulls,
Review comment:
this method is needed to use the relbuilder to create aggregates;
the overriden method is protected...and there is no way to access this level
of detail without exposing it
##
File path: ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist.q
##
@@ -0,0 +1,47 @@
+--! qt:transactional
+
+
+create table sketch_input (id int, category char(1))
+STORED AS ORC
+TBLPROPERTIES ('transactional'='true');
+
+insert into table sketch_input values
+ (1,'a'),(1, 'a'), (2, 'a'), (3, 'a'), (4, 'a'), (5, 'a'), (6, 'a'), (7,
'a'), (8, 'a'), (9, 'a'), (10, 'a'),
+ (6,'b'),(6, 'b'), (7, 'b'), (8, 'b'), (9, 'b'), (10, 'b'), (11, 'b'), (12,
'b'), (13, 'b'), (14, 'b'), (15, 'b')
+;
+
+select id,cume_dist() over (order by id) from sketch_input;
+
+select id,cume_dist() over (order by id),1.0-ds_kll_cdf(ds, CAST(-id AS FLOAT)
)[0]
Review comment:
these commands nicely show the original expression and the rewritten one
alongside to eachother - it would be nice to also add an assertion that they
are in the same neightbourhood
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java
##
@@ -368,4 +388,210 @@ void rewrite(AggregateCall aggCall) {
}
}
}
+
+ /**
+ * Generic support for rewriting Windowing expression into a different form
usually using joins.
+ */
+ private static abstract class WindowingToProjectAggregateJoinProject extends
RelOptRule {
+
+protected final String sketchType;
+
+public WindowingToProjectAggregateJoinProject(String sketchType) {
+ super(operand(HiveProject.class, any()), HiveRelFactories.HIVE_BUILDER,
null);
+ this.sketchType = sketchType;
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+
+ VbuilderPAP vb = buildProcessor(call);
+ RelNode newProject = vb.processProject(project);
+
+ if (newProject == project) {
+return;
+ } else {
+call.transformTo(newProject);
+ }
+}
+
+protected abstract VbuilderPAP buildProcessor(RelOptRuleCall call);
+
+protected static abstract class VbuilderPAP {
+ private final String sketchClass;
+ protected final RelBuilder relBuilder;
+ protected final RexBuilder rexBuilder;
+
+ protected VbuilderPAP(String sketchClass, RelBuilder relBuilder) {
+this.sketchClass = sketchClass;
+this.relBuilder = relBuilder;
+rexBuilder = relBuilder.getRexBuilder();
+ }
+
+
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Status: Patch Available (was: In Progress) > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Status: In Progress (was: Patch Available) > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Attachment: HIVE-23242.5.patch > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124809#comment-17124809
]
Hive QA commented on HIVE-23340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004686/HIVE-23340.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17094 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testCancelRenewTokenFlow
(batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testConnection (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValid (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValidNeg (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeProxyAuth (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testProxyAuth (batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testRenewDelegationToken
(batchId=236)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=236)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22725/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22725/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22725/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004686 - PreCommit-HIVE-Build
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124860#comment-17124860
]
Hive QA commented on HIVE-23242:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004687/HIVE-23242.5.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown_2]
(batchId=110)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testGetAllUrlsDirect
(batchId=220)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testGetAllUrlsZk
(batchId=220)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22726/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22726/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22726/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004687 - PreCommit-HIVE-Build
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch,
> HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch
>
>
> Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23525) TestAcidTxnCleanerService is unstable
[
https://issues.apache.org/jira/browse/HIVE-23525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Varga updated HIVE-23525:
---
Attachment: HIVE-23525.2.patch
> TestAcidTxnCleanerService is unstable
> -
>
> Key: HIVE-23525
> URL: https://issues.apache.org/jira/browse/HIVE-23525
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23525.1.patch, HIVE-23525.2.patch
>
>
> from time to time this exception happens
> http://34.66.156.144:8080/job/hive-c/7/console
> {code}
> 15:03:41 [INFO]
> 15:03:41 [INFO] ---
> 15:03:41 [INFO] T E S T S
> 15:03:41 [INFO] ---
> 15:03:42 [INFO] Running
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService
> 15:04:10 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 25.582 s <<< FAILURE! - in
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService
> 15:04:10 [ERROR]
> cleansAllCommittedTxns(org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService)
> Time elapsed: 9.952 s <<< FAILURE!
> 15:04:10 java.lang.AssertionError: expected:<6> but was:<7>
> 15:04:10 at
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService.cleansAllCommittedTxns(TestAcidTxnCleanerService.java:107)
> 15:04:10
> 15:04:10 [INFO]
> 15:04:10 [INFO] Results:
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Failures:
> 15:04:10 [ERROR] TestAcidTxnCleanerService.cleansAllCommittedTxns:107
> expected:<6> but was:<7>
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on
> project hive-standalone-metastore-server: There are test failures.
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] Please refer to
> /home/jenkins/agent/workspace/hive-c/standalone-metastore/metastore-server/target/surefire-reports
> for the individual test results.
> 15:04:10 [ERROR] Please refer to dump files (if any exist)
> [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
> 15:04:10 [ERROR] -> [Help 1]
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] To see the full stack trace of the errors, re-run Maven
> with the -e switch.
> 15:04:10 [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> 15:04:10 [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23525) TestAcidTxnCleanerService is unstable
[
https://issues.apache.org/jira/browse/HIVE-23525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Varga updated HIVE-23525:
---
Status: In Progress (was: Patch Available)
> TestAcidTxnCleanerService is unstable
> -
>
> Key: HIVE-23525
> URL: https://issues.apache.org/jira/browse/HIVE-23525
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23525.1.patch, HIVE-23525.2.patch
>
>
> from time to time this exception happens
> http://34.66.156.144:8080/job/hive-c/7/console
> {code}
> 15:03:41 [INFO]
> 15:03:41 [INFO] ---
> 15:03:41 [INFO] T E S T S
> 15:03:41 [INFO] ---
> 15:03:42 [INFO] Running
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService
> 15:04:10 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 25.582 s <<< FAILURE! - in
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService
> 15:04:10 [ERROR]
> cleansAllCommittedTxns(org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService)
> Time elapsed: 9.952 s <<< FAILURE!
> 15:04:10 java.lang.AssertionError: expected:<6> but was:<7>
> 15:04:10 at
> org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService.cleansAllCommittedTxns(TestAcidTxnCleanerService.java:107)
> 15:04:10
> 15:04:10 [INFO]
> 15:04:10 [INFO] Results:
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Failures:
> 15:04:10 [ERROR] TestAcidTxnCleanerService.cleansAllCommittedTxns:107
> expected:<6> but was:<7>
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> 15:04:10 [INFO]
> 15:04:10 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on
> project hive-standalone-metastore-server: There are test failures.
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] Please refer to
> /home/jenkins/agent/workspace/hive-c/standalone-metastore/metastore-server/target/surefire-reports
> for the individual test results.
> 15:04:10 [ERROR] Please refer to dump files (if any exist)
> [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
> 15:04:10 [ERROR] -> [Help 1]
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] To see the full stack trace of the errors, re-run Maven
> with the -e switch.
> 15:04:10 [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> 15:04:10 [ERROR]
> 15:04:10 [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> 15:04:10 [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124758#comment-17124758
]
Karen Coppage commented on HIVE-23533:
--
New method based on:
[https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/impl/OrcAcidUtils.java#L53-L69]
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch, HIVE-23533.02.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124802#comment-17124802
]
Karen Coppage commented on HIVE-23533:
--
:) Thank you!
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch, HIVE-23533.02.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23413) Create a new config to skip all locks
[ https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23413: --- Status: In Progress (was: Patch Available) > Create a new config to skip all locks > - > > Key: HIVE-23413 > URL: https://issues.apache.org/jira/browse/HIVE-23413 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23413.1.patch > > > From time-to-time some query is blocked on locks which should not. > To have a quick workaround for this we should have a config which the user > can set in the session to disable acquiring/checking locks, so we can provide > it immediately and then later investigate and fix the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23413) Create a new config to skip all locks
[ https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23413: --- Attachment: HIVE-23413.2.patch > Create a new config to skip all locks > - > > Key: HIVE-23413 > URL: https://issues.apache.org/jira/browse/HIVE-23413 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23413.1.patch, HIVE-23413.2.patch > > > From time-to-time some query is blocked on locks which should not. > To have a quick workaround for this we should have a config which the user > can set in the session to disable acquiring/checking locks, so we can provide > it immediately and then later investigate and fix the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23413) Create a new config to skip all locks
[ https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23413: --- Status: Patch Available (was: In Progress) > Create a new config to skip all locks > - > > Key: HIVE-23413 > URL: https://issues.apache.org/jira/browse/HIVE-23413 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23413.1.patch, HIVE-23413.2.patch > > > From time-to-time some query is blocked on locks which should not. > To have a quick workaround for this we should have a config which the user > can set in the session to disable acquiring/checking locks, so we can provide > it immediately and then later investigate and fix the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20890:
--
Status: Patch Available (was: Open)
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
> ++
> {code}
> So the per-partition SHARED_READ locks are no longer necessary, if the lock
> builder already includes the table-wide SHARED_READ locks.
> The removal of entire partitions is the only part which needs to be taken
> care of within this semantics as row-removal instead of directory removal
> (i.e "drop partition" -> "truncate partition" and have the truncation trigger
> a whole directory cleaner, so that the partition disappears when there are 0
> rows left).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-20890:
-
Assignee: Denys Kuzmenko (was: Zoltan Chovan)
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
> ++
> {code}
> So the per-partition SHARED_READ locks are no longer necessary, if the lock
> builder already includes the table-wide SHARED_READ locks.
> The removal of entire partitions is the only part which needs to be taken
> care of within this semantics as row-removal instead of directory removal
> (i.e "drop partition" -> "truncate partition" and have the truncation trigger
> a whole directory cleaner, so that the partition disappears when there are 0
> rows left).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-20890:
--
Attachment: HIVE-20890.1.patch
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
> ++
> {code}
> So the per-partition SHARED_READ locks are no longer necessary, if the lock
> builder already includes the table-wide SHARED_READ locks.
> The removal of entire partitions is the only part which needs to be taken
> care of within this semantics as row-removal instead of directory removal
> (i.e "drop partition" -> "truncate partition" and have the truncation trigger
> a whole directory cleaner, so that the partition disappears when there are 0
> rows left).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23244) Extract Create View analyzer from SemanticAnalyzer
[ https://issues.apache.org/jira/browse/HIVE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23244: -- Attachment: (was: HIVE-23244.07.patch) > Extract Create View analyzer from SemanticAnalyzer > -- > > Key: HIVE-23244 > URL: https://issues.apache.org/jira/browse/HIVE-23244 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23244.01.patch, HIVE-23244.02.patch, > HIVE-23244.03.patch, HIVE-23244.04.patch, HIVE-23244.05.patch, > HIVE-23244.06.patch, HIVE-23244.07.patch > > > Create View commands are not queries, but commands which have queries as a > part of them. Therefore a separate CreateViewAnalyzer is needed which uses > SemanticAnalyer to analyze it's query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23244) Extract Create View analyzer from SemanticAnalyzer
[ https://issues.apache.org/jira/browse/HIVE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23244: -- Attachment: HIVE-23244.07.patch > Extract Create View analyzer from SemanticAnalyzer > -- > > Key: HIVE-23244 > URL: https://issues.apache.org/jira/browse/HIVE-23244 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23244.01.patch, HIVE-23244.02.patch, > HIVE-23244.03.patch, HIVE-23244.04.patch, HIVE-23244.05.patch, > HIVE-23244.06.patch, HIVE-23244.07.patch > > > Create View commands are not queries, but commands which have queries as a > part of them. Therefore a separate CreateViewAnalyzer is needed which uses > SemanticAnalyer to analyze it's query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23591) When Worker fails to connect to metastore it should wait before retrying
[ https://issues.apache.org/jira/browse/HIVE-23591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23591: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the review [~lpinter]! > When Worker fails to connect to metastore it should wait before retrying > > > Key: HIVE-23591 > URL: https://issues.apache.org/jira/browse/HIVE-23591 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23591.02.patch, HIVE-23591.03.patch, > HIVE-23591.04.patch, HIVE-23591.patch > > > HIVE-23555 removed the sleep after Thrift exception. This would result in an > active loop if the HMS is unreachable. > We should wait before trying to connect again to the HMS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124779#comment-17124779
]
Hive QA commented on HIVE-23340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 496 unchanged - 32 fixed = 496 total (was 528) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22725/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22725/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22725/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22725/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Status: Patch Available (was: In Progress) > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Attachment: (was: HIVE-23242.5.patch) > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Status: In Progress (was: Patch Available) > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[ https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23242: --- Attachment: HIVE-23242.5.patch > Fix flaky tests testHouseKeepingThreadExistence in > TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader > - > > Key: HIVE-23242 > URL: https://issues.apache.org/jira/browse/HIVE-23242 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch, > HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch > > > Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for > details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: (was: HIVE-23340.7.patch) > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23340) TxnHandler cleanup
[ https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23340: --- Attachment: HIVE-23340.7.patch > TxnHandler cleanup > -- > > Key: HIVE-23340 > URL: https://issues.apache.org/jira/browse/HIVE-23340 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch, > HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch, > HIVE-23340.6.patch, HIVE-23340.7.patch > > > * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication > * Remove TxnStatus character constants and use the enum values -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124726#comment-17124726
]
Hive QA commented on HIVE-23533:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22724/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22724/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22724/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch, HIVE-23533.02.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124755#comment-17124755
]
Hive QA commented on HIVE-23533:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004683/HIVE-23533.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22724/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22724/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22724/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004683 - PreCommit-HIVE-Build
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch, HIVE-23533.02.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23462) Add option to rewrite CUME_DIST to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23462: Summary: Add option to rewrite CUME_DIST to sketch functions (was: Add option to rewrite NTILE to sketch functions) > Add option to rewrite CUME_DIST to sketch functions > --- > > Key: HIVE-23462 > URL: https://issues.apache.org/jira/browse/HIVE-23462 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch, > HIVE-23462.03.patch, HIVE-23462.04.patch > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23598) Add option to rewrite NTILE to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23598: --- > Add option to rewrite NTILE to sketch functions > --- > > Key: HIVE-23598 > URL: https://issues.apache.org/jira/browse/HIVE-23598 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23413) Create a new config to skip all locks
[
https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124904#comment-17124904
]
Hive QA commented on HIVE-23413:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004696/HIVE-23413.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17097 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22727/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22727/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22727/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004696 - PreCommit-HIVE-Build
> Create a new config to skip all locks
> -
>
> Key: HIVE-23413
> URL: https://issues.apache.org/jira/browse/HIVE-23413
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23413.1.patch, HIVE-23413.2.patch
>
>
> From time-to-time some query is blocked on locks which should not.
> To have a quick workaround for this we should have a config which the user
> can set in the session to disable acquiring/checking locks, so we can provide
> it immediately and then later investigate and fix the root cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23413) Create a new config to skip all locks
[ https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23413: --- Attachment: HIVE-23413.2.patch > Create a new config to skip all locks > - > > Key: HIVE-23413 > URL: https://issues.apache.org/jira/browse/HIVE-23413 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23413.1.patch, HIVE-23413.2.patch > > > From time-to-time some query is blocked on locks which should not. > To have a quick workaround for this we should have a config which the user > can set in the session to disable acquiring/checking locks, so we can provide > it immediately and then later investigate and fix the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23413) Create a new config to skip all locks
[ https://issues.apache.org/jira/browse/HIVE-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23413: --- Attachment: (was: HIVE-23413.2.patch) > Create a new config to skip all locks > - > > Key: HIVE-23413 > URL: https://issues.apache.org/jira/browse/HIVE-23413 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23413.1.patch, HIVE-23413.2.patch > > > From time-to-time some query is blocked on locks which should not. > To have a quick workaround for this we should have a config which the user > can set in the session to disable acquiring/checking locks, so we can provide > it immediately and then later investigate and fix the root cause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124794#comment-17124794
]
Peter Vary commented on HIVE-23533:
---
+1 like the version 2 even more
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch, HIVE-23533.02.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124830#comment-17124830
]
Hive QA commented on HIVE-23242:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 427 unchanged - 2 fixed = 427 total (was 429) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22726/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22726/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22726/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22726/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch,
> HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch
>
>
>
[jira] [Work logged] (HIVE-23462) Add option to rewrite CUME_DIST to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?focusedWorklogId=440747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440747
]
ASF GitHub Bot logged work on HIVE-23462:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 12:24
Start Date: 03/Jun/20 12:24
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1031:
URL: https://github.com/apache/hive/pull/1031#discussion_r434517408
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRules.java
##
@@ -368,4 +388,210 @@ void rewrite(AggregateCall aggCall) {
}
}
}
+
+ /**
+ * Generic support for rewriting Windowing expression into a different form
usually using joins.
+ */
+ private static abstract class WindowingToProjectAggregateJoinProject extends
RelOptRule {
+
+protected final String sketchType;
+
+public WindowingToProjectAggregateJoinProject(String sketchType) {
+ super(operand(HiveProject.class, any()), HiveRelFactories.HIVE_BUILDER,
null);
+ this.sketchType = sketchType;
+}
+
+@Override
+public void onMatch(RelOptRuleCall call) {
+ final Project project = call.rel(0);
+
+ VbuilderPAP vb = buildProcessor(call);
+ RelNode newProject = vb.processProject(project);
+
+ if (newProject == project) {
+return;
+ } else {
+call.transformTo(newProject);
+ }
+}
+
+protected abstract VbuilderPAP buildProcessor(RelOptRuleCall call);
+
+protected static abstract class VbuilderPAP {
+ private final String sketchClass;
+ protected final RelBuilder relBuilder;
+ protected final RexBuilder rexBuilder;
+
+ protected VbuilderPAP(String sketchClass, RelBuilder relBuilder) {
+this.sketchClass = sketchClass;
+this.relBuilder = relBuilder;
+rexBuilder = relBuilder.getRexBuilder();
+ }
+
+ final class ProcessShuttle extends RexShuttle {
+public RexNode visitOver(RexOver over) {
+ return processCall(over);
+}
+ };
+
+ protected final RelNode processProject(Project project) {
+RelNode origInput = project.getInput();
+relBuilder.push(origInput);
+RexShuttle shuttle = new ProcessShuttle();
+List newProjects = new ArrayList();
+for (RexNode expr : project.getChildExps()) {
+ newProjects.add(expr.accept(shuttle));
+}
+if (relBuilder.peek() == origInput) {
+ relBuilder.clear();
+ return project;
+}
+relBuilder.project(newProjects);
+return relBuilder.build();
+ }
+
+ private final RexNode processCall(RexNode expr) {
+if (expr instanceof RexOver) {
+ RexOver over = (RexOver) expr;
+ if (isApplicable(over)) {
+return rewrite(over);
+ }
+}
+return expr;
+ }
+
+ protected final SqlOperator getSqlOperator(String fnName) {
+UDFDescriptor fn =
DataSketchesFunctions.INSTANCE.getSketchFunction(sketchClass, fnName);
+if (!fn.getCalciteFunction().isPresent()) {
+ throw new RuntimeException(fn.toString() + " doesn't have a Calcite
function associated with it");
+}
+return fn.getCalciteFunction().get();
+ }
+
+ /**
+ * Do the rewrite for the given expression.
+ *
+ * When this method is invoked the {@link #relBuilder} will only contain
the current input.
+ * Expectation is to leave the new input there after the method finishes.
+ */
+ abstract RexNode rewrite(RexOver expr);
+
+ abstract boolean isApplicable(RexOver expr);
+
+}
+ }
+
+ public static class CumeDistRewrite extends
WindowingToProjectAggregateJoinProject {
+
+public CumeDistRewrite(String sketchType) {
+ super(sketchType);
+}
+
+@Override
+protected VbuilderPAP buildProcessor(RelOptRuleCall call) {
+ return new VB(sketchType, call.builder());
+}
+
+private static class VB extends VbuilderPAP {
+
+ protected VB(String sketchClass, RelBuilder relBuilder) {
+super(sketchClass, relBuilder);
+ }
+
+ @Override
+ boolean isApplicable(RexOver over) {
+SqlAggFunction aggOp = over.getAggOperator();
+RexWindow window = over.getWindow();
+if (aggOp.getName().equalsIgnoreCase("cume_dist") &&
window.orderKeys.size() == 1
+&& window.getLowerBound().isUnbounded() &&
window.getUpperBound().isUnbounded()) {
+ return true;
+}
+return false;
+ }
+
+ @Override
+ RexNode rewrite(RexOver over) {
+RexWindow w = over.getWindow();
+RexFieldCollation orderKey = w.orderKeys.get(0);
+// we don't really support nulls in aggregate/etc...they are actually
ignored
+// so some hack will
[jira] [Updated] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-20784: -- Parent: HIVE-22611 Issue Type: Sub-task (was: Improvement) > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: (was: HIVE-22681.1.patch) > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23595: Resolution: Fixed Status: Resolved (was: Patch Available) > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23595: Fix Version/s: 4.0.0 > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440911
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 18:13
Start Date: 03/Jun/20 18:13
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434761769
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
I poked around a bit, nothing is obviously reentrant. Do you know where
this is happening?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440911)
Time Spent: 1h 20m (was: 1h 10m)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23520) REPL: repl dump could add support for immutable dataset
[ https://issues.apache.org/jira/browse/HIVE-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125092#comment-17125092 ] Aasha Medhi commented on HIVE-23520: Is this both for managed and external tables? There is a Metadata only config for external tables which dumps partition, stats and metadata without dumping any data. Will that be useful? Can you please add a pull request where I can add the review comments > REPL: repl dump could add support for immutable dataset > --- > > Key: HIVE-23520 > URL: https://issues.apache.org/jira/browse/HIVE-23520 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23520.1.patch > > > Currently, "REPL DUMP" ends up copying entire dataset along with partition > information, stats etc in its dump folder. However, there are cases (e.g > large reference datasets), where we need a way to just retain metadata along > with partition information & stats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21141) Fix some spell errors in Hive
[ https://issues.apache.org/jira/browse/HIVE-21141?focusedWorklogId=440884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440884 ] ASF GitHub Bot logged work on HIVE-21141: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:47 Start Date: 03/Jun/20 16:47 Worklog Time Spent: 10m Work Description: belugabehr commented on a change in pull request #519: URL: https://github.com/apache/hive/pull/519#discussion_r434711070 ## File path: hcatalog/src/docs/src/documentation/skinconf.xml ## @@ -38,11 +38,11 @@ which will be used to configure the chosen Forrest skin. false true - + true > Key: HIVE-21141 > URL: https://issues.apache.org/jira/browse/HIVE-21141 > Project: Hive > Issue Type: Bug >Reporter: Bo Xu >Assignee: Bo Xu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21141.1.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Fix som spell errors in Hive -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23358) MSCK REPAIR should remove all insignificant zeroes from partition values (for numeric datatypes) before creating the partitions
[
https://issues.apache.org/jira/browse/HIVE-23358?focusedWorklogId=440891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440891
]
ASF GitHub Bot logged work on HIVE-23358:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 17:05
Start Date: 03/Jun/20 17:05
Worklog Time Spent: 10m
Work Description: adesh-rao commented on pull request #1050:
URL: https://github.com/apache/hive/pull/1050#issuecomment-638329732
@sankarh Can you take a look at the PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440891)
Time Spent: 20m (was: 10m)
> MSCK REPAIR should remove all insignificant zeroes from partition values (for
> numeric datatypes) before creating the partitions
> ---
>
> Key: HIVE-23358
> URL: https://issues.apache.org/jira/browse/HIVE-23358
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Adesh Kumar Rao
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For the following scenario
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1;
> {noformat}
> ++
> | partition |
> ++
> | year=2020/month=03/day=10 |
> | year=2020/month=03/day=11 |
> ++
> {noformat}
> 5.show table extended like 't1' partition (Year=2020, Month=03, Day=11);
> will throw an error:
> {noformat}
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10006]: Partition not found {year=2020, month=3, day=11}
> (state=42000,code=10006)
> {noformat}
> When the partition directory are created without the extra zeroes, this works
> fine.
> {noformat}
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=3/day=11
> {noformat}
> This happens because while searching for partitions, hive strips the extra
> "0" in month key and then queries the metastore
> (partSpec="year=2020/month=3/day=10") which returns no rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[ https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23516: --- Attachment: HIVE-23516.06.patch Status: Patch Available (was: In Progress) > Store hive replication policy execution metrics in the relational DB > > > Key: HIVE-23516 > URL: https://issues.apache.org/jira/browse/HIVE-23516 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch, > HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch, > HIVE-23516.06.patch, Replication Metrics.pdf > > Time Spent: 4.5h > Remaining Estimate: 0h > > Details documented in the attached doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23516) Store hive replication policy execution metrics in the relational DB
[ https://issues.apache.org/jira/browse/HIVE-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23516: --- Status: In Progress (was: Patch Available) > Store hive replication policy execution metrics in the relational DB > > > Key: HIVE-23516 > URL: https://issues.apache.org/jira/browse/HIVE-23516 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23516.01.patch, HIVE-23516.02.patch, > HIVE-23516.03.patch, HIVE-23516.04.patch, HIVE-23516.05.patch, > HIVE-23516.06.patch, Replication Metrics.pdf > > Time Spent: 4.5h > Remaining Estimate: 0h > > Details documented in the attached doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21790) Bump Java to 1.8
[
https://issues.apache.org/jira/browse/HIVE-21790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125184#comment-17125184
]
Hive QA commented on HIVE-21790:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12969644/0001-HIVE-21790-Update-to-Java-1.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22733/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22733/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22733/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:39.195
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22733/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:39.198
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
847673a..2795d32 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 847673a HIVE-22017: Keep HMS interfaces backward compatible with
changes for HIVE-21637 (Kishen Das, reviewed by Vihang Karajgaonkar)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:01:40.649
+ rm -rf ../yetus_PreCommit-HIVE-Build-22733
+ mkdir ../yetus_PreCommit-HIVE-Build-22733
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22733
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22733/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/README.md: does not exist in index
error: a/pom.xml: does not exist in index
Trying to apply the patch with -p1
error: patch failed: README.md:75
Falling back to three-way merge...
Applied patch to 'README.md' with conflicts.
error: patch failed: pom.xml:804
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: README.md:75
Falling back to three-way merge...
Applied patch to 'README.md' with conflicts.
error: patch failed: pom.xml:804
Falling back to three-way merge...
Applied patch to 'pom.xml' with conflicts.
U README.md
U pom.xml
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22733
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12969644 - PreCommit-HIVE-Build
> Bump Java to 1.8
>
>
> Key: HIVE-21790
> URL: https://issues.apache.org/jira/browse/HIVE-21790
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.5
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-21790-Update-to-Java-1.8.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We're using Hive for reading Parquet files, but we would like to move from
> gzip to zstandard compression. Currently, the Parquet support of Hive is old
> because we can't upgrade since Parquet is Java 1.8+. Therefore it is a good
> idea to upgrade Hive as well.
> GA support of Java 1.7 is also almost over:
>
[jira] [Commented] (HIVE-21895) Kafka Storage handler uses deprecated Kafka client methods
[
https://issues.apache.org/jira/browse/HIVE-21895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125187#comment-17125187
]
Hive QA commented on HIVE-21895:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12972262/HIVE-21895.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22734/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22734/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22734/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:58.860
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22734/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:58.863
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2795d32 HIVE-23595: Do not query task guaranteed status when wlm
off (Mustafa Iman via Gopal Vijayaraghavan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-06-03 18:02:59.910
+ rm -rf ../yetus_PreCommit-HIVE-Build-22734
+ mkdir ../yetus_PreCommit-HIVE-Build-22734
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22734
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22734/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:46
Falling back to three-way merge...
Applied patch to
'kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:46
Falling back to three-way merge...
Applied patch to
'kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java'
with conflicts.
U
kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22734
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12972262 - PreCommit-HIVE-Build
> Kafka Storage handler uses deprecated Kafka client methods
> --
>
> Key: HIVE-21895
> URL: https://issues.apache.org/jira/browse/HIVE-21895
> Project: Hive
> Issue Type: Improvement
> Components: kafka integration
>Affects Versions: 4.0.0
>Reporter: Kristopher Kane
>Assignee: Kristopher Kane
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21895.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Kafka client version is 2.2 and there are deprecated methods used like
> {code:java}
> producer.close(0, TimeUnit){code}
> in SimpleKafkaWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125242#comment-17125242
]
Hive QA commented on HIVE-22674:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004744/HIVE-22674.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17102 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[compute_stats_binary]
(batchId=75)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.allocateNextWriteIdRetriesAfterDetectingConflictingConcurrentInsert
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22735/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22735/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004744 - PreCommit-HIVE-Build
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?focusedWorklogId=440762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440762
]
ASF GitHub Bot logged work on HIVE-22201:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 13:02
Start Date: 03/Jun/20 13:02
Worklog Time Spent: 10m
Work Description: himanshu-mishra closed pull request #772:
URL: https://github.com/apache/hive/pull/772
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440762)
Time Spent: 20m (was: 10m)
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?focusedWorklogId=440763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440763
]
ASF GitHub Bot logged work on HIVE-22201:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 13:02
Start Date: 03/Jun/20 13:02
Worklog Time Spent: 10m
Work Description: himanshu-mishra commented on pull request #772:
URL: https://github.com/apache/hive/pull/772#issuecomment-638181669
Closing as code was merged. Ref:
[HIVE-22201](https://issues.apache.org/jira/browse/HIVE-22201)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440763)
Time Spent: 0.5h (was: 20m)
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[ https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23277: - Attachment: HIVE-23277.1.patch > HiveProtoLogger should carry out JSON conversion in its own thread > -- > > Key: HIVE-23277 > URL: https://issues.apache.org/jira/browse/HIVE-23277 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Minor > Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42 > AM.png > > > !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23602) Use Java Concurrent Package in Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23602: - > Use Java Concurrent Package in Operation Handle Set > --- > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23602) Use Java Concurrent Package for Operation Handle Set
[ https://issues.apache.org/jira/browse/HIVE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23602: -- Summary: Use Java Concurrent Package for Operation Handle Set (was: Use Java Concurrent Package in Operation Handle Set) > Use Java Concurrent Package for Operation Handle Set > > > Key: HIVE-23602 > URL: https://issues.apache.org/jira/browse/HIVE-23602 > Project: Hive > Issue Type: Bug >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125061#comment-17125061
]
Hive QA commented on HIVE-23340:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004710/HIVE-23340.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22730/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22730/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22730/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004710 - PreCommit-HIVE-Build
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[ https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22675: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~ashutoshc] for the review! > Replace Base64 in hive-standalone-metastore Package > --- > > Key: HIVE-22675 > URL: https://issues.apache.org/jira/browse/HIVE-22675 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch, > HIVE-22675.3.patch, HIVE-22675.3.patch, HIVE-22675.3.patch, > HIVE-22675.3.patch, HIVE-22675.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23596) Encode guaranteed task information in containerId
[ https://issues.apache.org/jira/browse/HIVE-23596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125151#comment-17125151 ] Gopal Vijayaraghavan commented on HIVE-23596: - LGTM - +1 Will add extra comments in the LlapTaskScheduler, since this takes advantage of arbitrary container-ids which have a full 64 bit range & do not get used outside of the runtime of the query (unlike say, task-ids, which show up in filenames). The LLAP daemon side doesn't care about a container-id, since we use a 1:1 correspondence between containers & tasks in LLAP. So a "tagged pointer" or "colored pointer" equivalent implementation is safe. > Encode guaranteed task information in containerId > - > > Key: HIVE-23596 > URL: https://issues.apache.org/jira/browse/HIVE-23596 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We should avoid calling LlapTaskScheduler to get initial isguaranteed flag > for all the tasks. It causes arbitrary delays in sending tasks out. Since > communicator is a single thread, any blocking there delays all the tasks. > There are [https://jira.apache.org/jira/browse/TEZ-4192] and > [https://jira.apache.org/jira/browse/HIVE-23589] for a proper solution to > this. However, that requires a Tez release which seems far right now. We can > replace the current hack with another hack that does not require locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22018) [ Interface changes ] Add table id to HMS get methods
[ https://issues.apache.org/jira/browse/HIVE-22018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das resolved HIVE-22018. --- Resolution: Duplicate This was done part of https://issues.apache.org/jira/browse/HIVE-22017 . > [ Interface changes ] Add table id to HMS get methods > - > > Key: HIVE-22018 > URL: https://issues.apache.org/jira/browse/HIVE-22018 > Project: Hive > Issue Type: Sub-task >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > > It is possible we remove a table and immediately move another table to occupy > the same name. CachedStore may retrieve the wrong table in this case. We > shall add tableid in every get_(table/partition) api, so we can compare the > one stored in TBLS (tableid is part of Table object) and check if the same > id, if not, HMS shall fail the read request. The initial table id can be > retrieved along with writeid (in DbTxnManager.getValidWriteIds call, to join > the TBLS table) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440899
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 17:35
Start Date: 03/Jun/20 17:35
Worklog Time Spent: 10m
Work Description: mythrocks commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434739477
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
I had left [a comment about this on
HIVE-19261](https://issues.apache.org/jira/browse/HIVE-19261?focusedCommentId=16929610=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16929610).
I think `makeInstance()` makes reentrant calls into
`InstanceCache::retrieve()`.
`computeIfAbsent()` expects that `cache` not be modified in the lambda.
Calling calling `makeInstance()` from `computeIfAbsent()` might lead to a
hang. I ran into this when testing an older version of this patch, some months
ago.
It's possible that the reentrant call no longer occurs. I'd check this
before using `computeIfAbsent()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440899)
Time Spent: 1h 10m (was: 1h)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21895) Kafka Storage handler uses deprecated Kafka client methods
[
https://issues.apache.org/jira/browse/HIVE-21895?focusedWorklogId=440873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440873
]
ASF GitHub Bot logged work on HIVE-21895:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 16:38
Start Date: 03/Jun/20 16:38
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #680:
URL: https://github.com/apache/hive/pull/680#issuecomment-638314315
Can you please take another crack at this?
Also, be mindful of this situation:
`
LOG.info("Flushing Kafka Producer with writerId [{}]", writerId);
producer.flush();
LOG.info("Closing WriterId [{}]", writerId);
producer.close();
`
I think the 'flush' should be wrapped in a try-finally block with the close.
Right now, if the flush fails, the close is skipped.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440873)
Time Spent: 20m (was: 10m)
> Kafka Storage handler uses deprecated Kafka client methods
> --
>
> Key: HIVE-21895
> URL: https://issues.apache.org/jira/browse/HIVE-21895
> Project: Hive
> Issue Type: Improvement
> Components: kafka integration
>Affects Versions: 4.0.0
>Reporter: Kristopher Kane
>Assignee: Kristopher Kane
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21895.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Kafka client version is 2.2 and there are deprecated methods used like
> {code:java}
> producer.close(0, TimeUnit){code}
> in SimpleKafkaWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125128#comment-17125128 ] Vihang Karajgaonkar commented on HIVE-22017: Patch merged into master. Thanks for your contribution [~kishendas] > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-22017: --- Target Version/s: (was: 2.3.7) > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23242) Fix flaky tests testHouseKeepingThreadExistence in TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
[
https://issues.apache.org/jira/browse/HIVE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125130#comment-17125130
]
Hive QA commented on HIVE-23242:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004711/HIVE-23242.5.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22731/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22731/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22731/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004711 - PreCommit-HIVE-Build
> Fix flaky tests testHouseKeepingThreadExistence in
> TestMetastoreHousekeepingLeaderEmptyConfig and TestMetastoreHousekeepingLeader
> -
>
> Key: HIVE-23242
> URL: https://issues.apache.org/jira/browse/HIVE-23242
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23242.1.patch, HIVE-23242.2.patch,
> HIVE-23242.3.patch, HIVE-23242.4.patch, HIVE-23242.5.patch
>
>
> Tests were ignored, see https://issues.apache.org/jira/browse/HIVE-23221 for
> details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22017) [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637
[ https://issues.apache.org/jira/browse/HIVE-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-22017: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > [ Interface changes ] Keep HMS interfaces backward compatible with changes > for HIVE-21637 > - > > Key: HIVE-22017 > URL: https://issues.apache.org/jira/browse/HIVE-22017 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.7 >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22017.1.patch, HIVE-22017.2.patch, > HIVE-22017.3.patch, HIVE-22017.4.patch, HIVE-22017.5.patch, > HIVE-22017.6.patch, HIVE-22017.7.patch, HIVE-22017.8.patch > > > As part of HIVE-21637 we would have to introduce ValidWriteIdList in several > methods. Also, in the long term, we should deprecate and remove all the > methods that take direct arguments, as the service definition keeps changing > whenever we add/remove arguments, making it hard to maintain backward > compatibility. So, instead of adding writeId in bunch of get_xxx calls that > take direct arguments, we will create new set of methods that take Request > object and return Response object. We shall mark those deprecated and remove > in future version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?focusedWorklogId=440886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440886 ] ASF GitHub Bot logged work on HIVE-20784: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:51 Start Date: 03/Jun/20 16:51 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #451: URL: https://github.com/apache/hive/pull/451#issuecomment-638321777 Nice fix. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440886) Remaining Estimate: 0h Time Spent: 10m > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23595) Do not query task guaranteed status when wlm off
[ https://issues.apache.org/jira/browse/HIVE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125146#comment-17125146 ] László Bodor commented on HIVE-23595: - pushed to master, thanks [~mustafaiman] for the patch and [~gopalv] for the review! > Do not query task guaranteed status when wlm off > > > Key: HIVE-23595 > URL: https://issues.apache.org/jira/browse/HIVE-23595 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23595.patch > > > LlapTaskCommunicator queries scheduler for every task guaranteed status. When > workload management is off it is always false. There is no need for the > synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23569) [RawStore] RawStore changes to facilitate HMS cache consistency
[ https://issues.apache.org/jira/browse/HIVE-23569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das reassigned HIVE-23569: - Assignee: (was: Kishen Das) > [RawStore] RawStore changes to facilitate HMS cache consistency > > > Key: HIVE-23569 > URL: https://issues.apache.org/jira/browse/HIVE-23569 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Priority: Major > > ObjectStore should use additional fields tableId and validWriteIdList fields > for all read methods to compare with cached ValidWriteIdList. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (HIVE-23569) [RawStore] RawStore changes to facilitate HMS cache consistency
[ https://issues.apache.org/jira/browse/HIVE-23569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23569 stopped by Kishen Das. - > [RawStore] RawStore changes to facilitate HMS cache consistency > > > Key: HIVE-23569 > URL: https://issues.apache.org/jira/browse/HIVE-23569 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > ObjectStore should use additional fields tableId and validWriteIdList fields > for all read methods to compare with cached ValidWriteIdList. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-20784) Migrate hbase.util.Base64 to java.util.Base64
[ https://issues.apache.org/jira/browse/HIVE-20784?focusedWorklogId=440887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440887 ] ASF GitHub Bot logged work on HIVE-20784: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:52 Start Date: 03/Jun/20 16:52 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #451: URL: https://github.com/apache/hive/pull/451#issuecomment-638321924 @kgyrtkirk Please assist :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440887) Time Spent: 20m (was: 10m) > Migrate hbase.util.Base64 to java.util.Base64 > - > > Key: HIVE-20784 > URL: https://issues.apache.org/jira/browse/HIVE-20784 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.1.0 > Environment: HBase 2.0.2 > Hive 3.1.0 >Reporter: Dagang Wei >Assignee: Dagang Wei >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > By default Hive 3.1.0 depends on HBase 2.0.0-alpha4. HBase 2.0.2 migrated > from hbase.util.Base64 to java.util.Base64 (HBASE-20884), which causes Hive > 3.1.0 fails to build with HBase 2.0.2. > > $ cd hbase-handler > $ mvn package -DskipTests -Dhbase.version=2.0.2 > [ERROR] > .../hive/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[29,36] > cannot find symbol > [ERROR] symbol: class Base64 > [ERROR] location: package org.apache.hadoop.hbase.util > > To make Hive works with 2.0.2+ (and also older versions), we should consider > migrating Hive to java.util.Base64. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22674) Replace Base64 in serde Package
[ https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22674: -- Attachment: HIVE-22674.2.patch > Replace Base64 in serde Package > --- > > Key: HIVE-22674 > URL: https://issues.apache.org/jira/browse/HIVE-22674 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch, > HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: (was: HIVE-22681.1.patch) > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22681) Replace Base64 in hcatalog-webhcat Package
[ https://issues.apache.org/jira/browse/HIVE-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22681: -- Attachment: HIVE-22681.2.patch > Replace Base64 in hcatalog-webhcat Package > -- > > Key: HIVE-22681 > URL: https://issues.apache.org/jira/browse/HIVE-22681 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22681.1.patch, HIVE-22681.1.patch, > HIVE-22681.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125152#comment-17125152
]
Hive QA commented on HIVE-23277:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 18 new + 10 unchanged - 0
fixed = 28 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22732/dev-support/hive-personality.sh
|
| git revision | master / 64b8c0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22732/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22732/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[
https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125182#comment-17125182
]
Hive QA commented on HIVE-23277:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004724/HIVE-23277.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17103 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22732/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22732/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22732/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004724 - PreCommit-HIVE-Build
> HiveProtoLogger should carry out JSON conversion in its own thread
> --
>
> Key: HIVE-23277
> URL: https://issues.apache.org/jira/browse/HIVE-23277
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Attila Magyar
>Priority: Minor
> Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42
> AM.png
>
>
> !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22674) Replace Base64 in serde Package
[
https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125212#comment-17125212
]
Hive QA commented on HIVE-22674:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} serde: The patch generated 1 new + 33 unchanged - 1
fixed = 34 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22735/dev-support/hive-personality.sh
|
| git revision | master / 2795d32 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22735/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22735/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace Base64 in serde Package
> ---
>
> Key: HIVE-22674
> URL: https://issues.apache.org/jira/browse/HIVE-22674
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch,
> HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23244) Extract Create View analyzer from SemanticAnalyzer
[
https://issues.apache.org/jira/browse/HIVE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124930#comment-17124930
]
Hive QA commented on HIVE-23244:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1523 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} ql: The patch generated 2 new + 532 unchanged - 7
fixed = 534 total (was 539) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 1 new + 1522 unchanged - 1 fixed = 1523
total (was 1523) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field org.apache.hadoop.hive.ql.plan.LoadFileDesc.createViewDesc is
transient but isn't set by deserialization In LoadFileDesc.java:but isn't set
by deserialization In LoadFileDesc.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22728/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22728/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22728/yetus/new-findbugs-ql.html
|
| modules | C: common parser ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22728/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extract Create View analyzer from SemanticAnalyzer
> --
>
> Key: HIVE-23244
> URL: https://issues.apache.org/jira/browse/HIVE-23244
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23244.01.patch, HIVE-23244.02.patch,
> HIVE-23244.03.patch, HIVE-23244.04.patch, HIVE-23244.05.patch,
>
[jira] [Commented] (HIVE-23244) Extract Create View analyzer from SemanticAnalyzer
[
https://issues.apache.org/jira/browse/HIVE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124948#comment-17124948
]
Hive QA commented on HIVE-23244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004709/HIVE-23244.07.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22728/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22728/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22728/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004709 - PreCommit-HIVE-Build
> Extract Create View analyzer from SemanticAnalyzer
> --
>
> Key: HIVE-23244
> URL: https://issues.apache.org/jira/browse/HIVE-23244
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23244.01.patch, HIVE-23244.02.patch,
> HIVE-23244.03.patch, HIVE-23244.04.patch, HIVE-23244.05.patch,
> HIVE-23244.06.patch, HIVE-23244.07.patch
>
>
> Create View commands are not queries, but commands which have queries as a
> part of them. Therefore a separate CreateViewAnalyzer is needed which uses
> SemanticAnalyer to analyze it's query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440815 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:58 Start Date: 03/Jun/20 14:58 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error and there is no way to push the size back into the stream. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully shutdown the connection. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440815) Time Spent: 2h 40m (was: 2.5h) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440847
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:50
Start Date: 03/Jun/20 15:50
Worklog Time Spent: 10m
Work Description: Shivamohan07 edited a comment on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638285159
> Please close. Committed to Hive master through JIRA. Good fix!
Thanks!, Closing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440847)
Time Spent: 1h (was: 50m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440845
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:50
Start Date: 03/Jun/20 15:50
Worklog Time Spent: 10m
Work Description: Shivamohan07 commented on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638285159
Thanks
> Please close. Committed to Hive master through JIRA. Good fix!
Thanks!, Closing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440845)
Time Spent: 50m (was: 40m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23026) Support add a yarn application name for tez on hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-23026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23026: -- Labels: pull-request-available (was: ) > Support add a yarn application name for tez on hiveserver2 > -- > > Key: HIVE-23026 > URL: https://issues.apache.org/jira/browse/HIVE-23026 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.3.5 >Reporter: Jake Xie >Priority: Major > Labels: pull-request-available > Fix For: 2.3.5 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently tez on hiveServer2 cannot specify yarn application name, which is > not very convenient for locating the problem SQL, so i added a configuration > item to support setting tez job name -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23026) Support add a yarn application name for tez on hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-23026?focusedWorklogId=440833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440833
]
ASF GitHub Bot logged work on HIVE-23026:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:21
Start Date: 03/Jun/20 15:21
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #947:
URL: https://github.com/apache/hive/pull/947#discussion_r434641680
##
File path: .travis.yml
##
@@ -42,4 +42,4 @@ before_install:
install: true
-script: travis_wait 30 mvn clean install -DskipTests -T 4 -q -Pitests
+script: mvn clean install -DskipTests -q -Pitests
Review comment:
Not related to this task. Please remove.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
##
@@ -310,7 +311,11 @@ protected void openInternal(final HiveConf conf,
Collection additionalFi
setupSessionAcls(tezConfig, conf);
-final TezClient session = TezClient.newBuilder("HIVE-" + sessionId,
tezConfig)
+String tezJobName = HiveConf.getVar(conf,ConfVars.HIVETEZJOBNAME);
+if (StringUtils.isBlank(tezJobName)){
+ tezJobName = "HIVE-" + sessionId;
Review comment:
What is the value of setting a hard coded name here?
I think it would be more useful if it would accept a Format String:
`
tezJobName = String.format("HIVE-%s", sessionId);
`
The default value of the configuration should be `HIVE-%s`
##
File path: .travis.yml
##
@@ -33,7 +33,7 @@ cache:
env:
MAVEN_SKIP_RC=true
- MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M"
Review comment:
Not related to this task. Please remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440833)
Remaining Estimate: 0h
Time Spent: 10m
> Support add a yarn application name for tez on hiveserver2
> --
>
> Key: HIVE-23026
> URL: https://issues.apache.org/jira/browse/HIVE-23026
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.3.5
>Reporter: Jake Xie
>Priority: Major
> Fix For: 2.3.5
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently tez on hiveServer2 cannot specify yarn application name, which is
> not very convenient for locating the problem SQL, so i added a configuration
> item to support setting tez job name
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440819 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 15:00 Start Date: 03/Jun/20 15:00 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error and there is no way to push the size back into the stream. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully close the Hive session on the server. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440819) Time Spent: 2h 50m (was: 2h 40m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23277) HiveProtoLogger should carry out JSON conversion in its own thread
[ https://issues.apache.org/jira/browse/HIVE-23277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-23277: - Status: Patch Available (was: Open) > HiveProtoLogger should carry out JSON conversion in its own thread > -- > > Key: HIVE-23277 > URL: https://issues.apache.org/jira/browse/HIVE-23277 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Minor > Attachments: HIVE-23277.1.patch, Screenshot 2020-04-23 at 11.27.42 > AM.png > > > !Screenshot 2020-04-23 at 11.27.42 AM.png|width=623,height=423! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23598) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124966#comment-17124966
]
Zoltan Haindrich commented on HIVE-23598:
-
I think this could be rewritten to be use 'cume_dist' and let the other rewrite
do the heavy lifting...
{code}
select id,ntile(4) over (order by id),CEIL(cume_dist() over (order by id)*4)
from sketch_input
order by id;
{code}
the NTILE udf has an easy advantage: it may return different values for the
same input value.
{code}
select id,ntile(2) over (order by id) from TABLE(VALUES(1),(1),(1),(1)) AS
t(id);
+-+-+
| id | ntile_window_0 |
+-+-+
| 1 | 1 |
| 1 | 1 |
| 1 | 2 |
| 1 | 2 |
+-+-+
{code}
> Add option to rewrite NTILE to sketch functions
> ---
>
> Key: HIVE-23598
> URL: https://issues.apache.org/jira/browse/HIVE-23598
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23601) Hive Statement Does Not Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23601:
-
> Hive Statement Does Not Clear Statement Handle on Error
> ---
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23601) Hive Statement Clear Statement Handle on Error
[
https://issues.apache.org/jira/browse/HIVE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23601:
--
Summary: Hive Statement Clear Statement Handle on Error (was: Hive
Statement Does Not Clear Statement Handle on Error)
> Hive Statement Clear Statement Handle on Error
> --
>
> Key: HIVE-23601
> URL: https://issues.apache.org/jira/browse/HIVE-23601
> Project: Hive
> Issue Type: Bug
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:java}
> private void closeStatementIfNeeded() throws SQLException {
> try {
> if (stmtHandle != null) {
> TCloseOperationReq closeReq = new TCloseOperationReq(stmtHandle);
> TCloseOperationResp closeResp = client.CloseOperation(closeReq);
> Utils.verifySuccessWithInfo(closeResp.getStatus());
> stmtHandle = null;
> }
> } catch (SQLException e) {
> throw e;
> } catch (Exception e) {
> throw new SQLException("Failed to close statement", "08S01", e);
> }
> }
> void closeClientOperation() throws SQLException {
> closeStatementIfNeeded();
> isQueryClosed = true;
> stmtHandle = null;
> }
> {code}
> {{verifySuccessWithInfo}} throws an {{Exception}} if it finds an error code
> and therefore leapfrogs over setting the statement handle to null (twice).
> Probably not what is intended since the original author(s) are tried twice to
> null it out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks
[
https://issues.apache.org/jira/browse/HIVE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125006#comment-17125006
]
Hive QA commented on HIVE-20890:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004704/HIVE-20890.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 17094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats3]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats4]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_table_stats]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explain_locks]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=94)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=273)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=274)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testForParallelBootstrapLoad
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testMetadataOnlyDump
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testNonParallelBootstrapLoad
(batchId=182)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testRetryFailure
(batchId=182)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testAllPartitions
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testNoStatsUpdateForSimpleReplTable
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testNoStatsUpdateForTxnReplTable
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testParallelOps
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionSubset
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionsWithDifferentColsAll
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testPartitionsWithDifferentColsExistingOnly
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testQueueingWithThreads
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnDynamicPartitions
(batchId=257)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnPartitions
(batchId=257)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
(batchId=178)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation (batchId=219)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22729/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22729/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22729/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004704 - PreCommit-HIVE-Build
> ACID: Allow whole table ReadLocks to skip all partition locks
> -
>
> Key: HIVE-20890
> URL: https://issues.apache.org/jira/browse/HIVE-20890
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20890.1.patch
>
>
> HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any
> SHARED_READ locks for read operations - in the presence of that lock, the
> insert overwrite no longer takes an exclusive lock.
> The only exclusive operation will be a schema change or drop table, which
> should take an exclusive lock on the entire table directly.
> {code}
> explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where
> ss_sold_date_sk=2452626
> ++
> | Explain |
> ++
> | LOCK INFORMATION: |
> | tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
> | tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 ->
> SHARED_READ |
>
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440840
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:46
Start Date: 03/Jun/20 15:46
Worklog Time Spent: 10m
Work Description: belugabehr edited a comment on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638282682
Please close. Committed to Hive master through JIRA. Good fix!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440840)
Time Spent: 0.5h (was: 20m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?focusedWorklogId=440838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440838
]
ASF GitHub Bot logged work on HIVE-22771:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 15:45
Start Date: 03/Jun/20 15:45
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #889:
URL: https://github.com/apache/hive/pull/889#issuecomment-638282682
Please close. Committed to Hive master through JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440838)
Time Spent: 20m (was: 10m)
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch,
> HIVE-22771.4.patch, HIVE-22771.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d
> .?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[
https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440851
]
ASF GitHub Bot logged work on HIVE-19261:
-
Author: ASF GitHub Bot
Created on: 03/Jun/20 16:01
Start Date: 03/Jun/20 16:01
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #807:
URL: https://github.com/apache/hive/pull/807#discussion_r434678396
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
Review comment:
Please change this (and the rest) to:
`
LOG.debug("Checking for hv: {}", hv);
`
http://www.slf4j.org/faq.html#logging_performance
##
File path: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
##
@@ -51,20 +51,19 @@ public Instance retrieve(SeedObject hv) throws
AvroSerdeException {
* Retrieve (or create if it doesn't exist) the correct Instance for this
* SeedObject using 'seenSchemas' to resolve circular references
*/
- public synchronized Instance retrieve(SeedObject hv,
- Set seenSchemas) throws AvroSerdeException {
+ public Instance retrieve(SeedObject hv, Set seenSchemas)
+throws AvroSerdeException {
if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString());
if(cache.containsKey(hv)) {
Review comment:
Please leverage `Map#computeIfAbsent` here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 440851)
Time Spent: 50m (was: 40m)
> Avro SerDe's InstanceCache should not be synchronized on retrieve
> -
>
> Key: HIVE-19261
> URL: https://issues.apache.org/jira/browse/HIVE-19261
> Project: Hive
> Issue Type: Improvement
>Reporter: Fangshi Li
>Assignee: Alexey Diomin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19261.1.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In HIVE-16175, upstream made a patch to fix the thread safety issue in
> AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache
> synchronized. While it should make InstanceCache thread-safe, making retrieve
> synchronized for the cache can be expensive in highly concurrent environment
> like Spark, as multiple threads need to be synchronized on entering the
> entire retrieve method.
> We are proposing another way to fix this thread safety issue by making the
> underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use
> atomic computeIfAbsent in the retrieve method to avoid synchronizing the
> entire method.
> While computeIfAbsent is only available on java 8 and java 7 is still
> supported in Hive,
> we use a pattern to simulate the behavior of computeIfAbsent. In the future,
> we should move to computeIfAbsent when Hive requires java 8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-19261) Avro SerDe's InstanceCache should not be synchronized on retrieve
[ https://issues.apache.org/jira/browse/HIVE-19261?focusedWorklogId=440852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440852 ] ASF GitHub Bot logged work on HIVE-19261: - Author: ASF GitHub Bot Created on: 03/Jun/20 16:01 Start Date: 03/Jun/20 16:01 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #807: URL: https://github.com/apache/hive/pull/807#issuecomment-638292070 Hive project is testing out using GitHub now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440852) Time Spent: 1h (was: 50m) > Avro SerDe's InstanceCache should not be synchronized on retrieve > - > > Key: HIVE-19261 > URL: https://issues.apache.org/jira/browse/HIVE-19261 > Project: Hive > Issue Type: Improvement >Reporter: Fangshi Li >Assignee: Alexey Diomin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19261.1.patch > > Time Spent: 1h > Remaining Estimate: 0h > > In HIVE-16175, upstream made a patch to fix the thread safety issue in > AvroSerDe's InstanceCache. This fix made the retrieve method in InstanceCache > synchronized. While it should make InstanceCache thread-safe, making retrieve > synchronized for the cache can be expensive in highly concurrent environment > like Spark, as multiple threads need to be synchronized on entering the > entire retrieve method. > We are proposing another way to fix this thread safety issue by making the > underlying map of InstanceCache as ConcurrentHashMap. Ideally, we can use > atomic computeIfAbsent in the retrieve method to avoid synchronizing the > entire method. > While computeIfAbsent is only available on java 8 and java 7 is still > supported in Hive, > we use a pattern to simulate the behavior of computeIfAbsent. In the future, > we should move to computeIfAbsent when Hive requires java 8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23340) TxnHandler cleanup
[
https://issues.apache.org/jira/browse/HIVE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125025#comment-17125025
]
Hive QA commented on HIVE-23340:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 496 unchanged - 32 fixed = 496 total (was 528) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22730/dev-support/hive-personality.sh
|
| git revision | master / c886653 |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22730/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TxnHandler cleanup
> --
>
> Key: HIVE-23340
> URL: https://issues.apache.org/jira/browse/HIVE-23340
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Attachments: HIVE-23340.1.patch, HIVE-23340.2.patch,
> HIVE-23340.3.patch, HIVE-23340.4.patch, HIVE-23340.5.patch,
> HIVE-23340.6.patch, HIVE-23340.7.patch
>
>
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
> * Remove TxnStatus character constants and use the enum values
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?focusedWorklogId=440814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-440814 ] ASF GitHub Bot logged work on HIVE-23526: - Author: ASF GitHub Bot Created on: 03/Jun/20 14:57 Start Date: 03/Jun/20 14:57 Worklog Time Spent: 10m Work Description: belugabehr edited a comment on pull request #1029: URL: https://github.com/apache/hive/pull/1029#issuecomment-638240582 As I understand it, the code calling the JDBC Driver captures and handles the OOM error and then closes the JDBC Driver. When it closes, it throws another error " CloseOperation failed: out of sequence response". You would like for the close to be silent here. Is that correct? If so, there's not a ton that can be done here (other than perhaps wrap the OOM error in a Thrift Exception). In this particular case, Thrift client is reading the size of the string off the wire, then trying to create a buffer large enough to read the string. This is where it fails. When it fails here, it loses its place in the stream. The data is still in the stream, but the size is lost since that was read before the OOM error. At this point, Thrift client is hosed and it can't read anything else. When the JDBC Driver is closed, it sends a "close" request to HS2 to gracefully shutdown the connection. However, it is unable to read the response properly because, again, it lost its place and does not know how to find the next message. I'm not sure what the fix is here, but I don't think this is it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 440814) Time Spent: 2.5h (was: 2h 20m) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 2.5h > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage message to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
