[
https://issues.apache.org/jira/browse/HIVE-23725?focusedWorklogId=450282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-450282
]
ASF GitHub Bot logged work on HIVE-23725:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jun/20 07:30
Start Date: 24/Jun/20 07:30
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1151:
URL: https://github.com/apache/hive/pull/1151#discussion_r444699013
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionRetryLockPlugin.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.reexec;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.plan.mapper.PlanMapper;
+import org.apache.hadoop.hive.ql.processors.CommandProcessorException;
+
+public class ReExecutionRetryLockPlugin implements IReExecutionPlugin {
+
+ private Driver coreDriver;
+ private int maxRetryLockExecutions = 1;
+
+ @Override
+ public void initialize(Driver driver) {
+ coreDriver = driver;
+ maxRetryLockExecutions = 1 +
coreDriver.getConf().getIntVar(HiveConf.ConfVars.HIVE_QUERY_MAX_REEXECUTION_RETRYLOCK_COUNT);
+ }
+
+ @Override
+ public void beforeExecute(int executionIndex, boolean explainReOptimization)
{
+ }
+
+ @Override
+ public boolean shouldReExecute(int executionNum, CommandProcessorException
ex) {
+ return executionNum < maxRetryLockExecutions && ex != null &&
+
ex.getMessage().contains(Driver.SNAPSHOT_WAS_OUTDATED_WHEN_LOCKS_WERE_ACQUIRED);
+ }
+
+ @Override
+ public void prepareToReExecute() {
+ }
+
+ @Override
+ public boolean shouldReExecute(int executionNum, PlanMapper oldPlanMapper,
PlanMapper newPlanMapper) {
+ return executionNum < maxRetryLockExecutions;
Review comment:
The ReExecutePluginInterface is not really straightforward. This method
is called after the query is recompiled from the reExecDriver
shouldReExecuteAfterCompile . Basicly it asks after the recompilation do you
want to continue the execution. It is only used by the reoptimize plugin, which
tries to recompile the query with different statistics (if I understand
correctly) and only reexecutes if the plan did change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 450282)
Time Spent: 4h 10m (was: 4h)
> ValidTxnManager snapshot outdating causing partial reads in merge insert
> ------------------------------------------------------------------------
>
> Key: HIVE-23725
> URL: https://issues.apache.org/jira/browse/HIVE-23725
> Project: Hive
> Issue Type: Bug
> Reporter: Peter Varga
> Assignee: Peter Varga
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When the ValidTxnManager invalidates the snapshot during merge insert and
> starts to read committed transactions that were not committed when the query
> compilation happened, it can cause partial read problems if the committed
> transaction created new partition in the source or target table.
> The solution should be not only fix the snapshot but also recompile the query
> and acquire the locks again.
> You could construct an example like this:
> 1. open and compile transaction 1 that merge inserts data from a partitioned
> source table that has a few partition.
> 2. Open, run and commit transaction 2 that inserts data to an old and a new
> partition to the source table.
> 3. Open, run and commit transaction 3 that inserts data to the target table
> of the merge statement, that will retrigger a snapshot generation in
> transaction 1.
> 4. Run transaction 1, the snapshot will be regenerated, and it will read
> partial data from transaction 2 breaking the ACID properties.
> Different setup.
> Switch the transaction order:
> 1. compile transaction 1 that inserts data to an old and a new partition of
> the source table.
> 2. compile transaction 2 that insert data to the target table
> 2. compile transaction 3 that merge inserts data from the source table to the
> target table
> 3. run and commit transaction 1
> 4. run and commit transaction 2
> 5. run transaction 3, since it cointains 1 and 2 in its snaphot the
> isValidTxnListState will be triggered and we do a partial read of the
> transaction 1 for the same reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
