dzcxzl created HIVE-30265:
-----------------------------
Summary: Table rename retried by RetryingHMSHandler leaves
partitions at the old location
Key: HIVE-30265
URL: https://issues.apache.org/jira/browse/HIVE-30265
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: dzcxzl
h3. Problem
When {{alter_table}} renames a managed partitioned table and the first attempt
fails with a recoverable datastore error, {{RetryingHMSHandler}} retries the
call and the retry reports success, but the table is left inconsistent:
* table name: new
* table location: new path, which does not exist on the file system
* partition locations: still the old path
* data: still in the old directory
The client gets no error.
h3. Root cause
{{HiveAlterHandler.alterTable}} decides whether a rename has to move data by
comparing the locations of the old and the new table object:
{code:java}
boolean renamedManagedTable = rename &&
!oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
&& (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation()) == 0
|| StringUtils.isEmpty(newt.getSd().getLocation()))
&& (!MetaStoreUtils.isExternalTable(oldt));
{code}
Inside that flow it rewrites the location of the caller's object:
{code:java}
destPath = new Path(databasePath, newTblName);
newt.getSd().setLocation(destPath.toString());
{code}
{{RetryingHMSHandler}} retries with the same argument objects
({{{}method.invoke(baseHandler, args){}}}). If the first attempt fails after
this point with an exception that {{DatabaseProduct.isRecoverableException}}
accepts, the following happens:
# The first attempt rolls back the transaction and moves the directory back to
the old path.
# The retry sees {{newt}} with the location already set to the renamed path,
so {{renamedManagedTable}} is false.
# The retry is handled as "rename with a location chosen by the caller": only
the table name and table location are written, and the transaction commits.
h3. Impact
Queries on the renamed table keep working because partition locations still
point at existing data, so the inconsistency goes unnoticed. The old directory
stays in use. If another table is later created with, or renamed to, the old
name, both tables share the same partition directories. Dropping the renamed
managed table then deletes those directories, because its partition locations
are outside its own table directory, and the other table loses its data.
h3. How to reproduce
# Create a managed partitioned table with at least one partition.
# Make the first {{RawStore.alterPartitions}} call fail with a
{{{}JDODataStoreException{}}}.
# Rename the table through the Thrift API with
{{hive.metastore.hmshandler.retry.attempts}} > 1.
The call succeeds, the new table directory does not exist, and the partitions
still point at the old directory. A unit test doing exactly this with
{{InjectableBehaviourObjectStore}} is included in the patch.
h3. Proposed fix
Remember the location passed in by the caller before rewriting it, and put it
back when the transaction does not commit, so that a retry sees the same input
as the first attempt.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)