dzcxzl created HIVE-30265:
-----------------------------

             Summary: Table rename retried by RetryingHMSHandler leaves 
partitions at the old location
                 Key: HIVE-30265
                 URL: https://issues.apache.org/jira/browse/HIVE-30265
             Project: Hive
          Issue Type: Bug
          Components: Metastore
            Reporter: dzcxzl


h3. Problem

When {{alter_table}} renames a managed partitioned table and the first attempt 
fails with a recoverable datastore error, {{RetryingHMSHandler}} retries the 
call and the retry reports success, but the table is left inconsistent:
 * table name: new
 * table location: new path, which does not exist on the file system
 * partition locations: still the old path
 * data: still in the old directory

The client gets no error.
h3. Root cause

{{HiveAlterHandler.alterTable}} decides whether a rename has to move data by 
comparing the locations of the old and the new table object:
{code:java}
boolean renamedManagedTable = rename && 
!oldt.getTableType().equals(TableType.VIRTUAL_VIEW.toString())
    && (oldt.getSd().getLocation().compareTo(newt.getSd().getLocation()) == 0
        || StringUtils.isEmpty(newt.getSd().getLocation()))
    && (!MetaStoreUtils.isExternalTable(oldt));
{code}
Inside that flow it rewrites the location of the caller's object:
{code:java}
destPath = new Path(databasePath, newTblName);
newt.getSd().setLocation(destPath.toString());
{code}
{{RetryingHMSHandler}} retries with the same argument objects 
({{{}method.invoke(baseHandler, args){}}}). If the first attempt fails after 
this point with an exception that {{DatabaseProduct.isRecoverableException}} 
accepts, the following happens:
 # The first attempt rolls back the transaction and moves the directory back to 
the old path.
 # The retry sees {{newt}} with the location already set to the renamed path, 
so {{renamedManagedTable}} is false.
 # The retry is handled as "rename with a location chosen by the caller": only 
the table name and table location are written, and the transaction commits.

h3. Impact

Queries on the renamed table keep working because partition locations still 
point at existing data, so the inconsistency goes unnoticed. The old directory 
stays in use. If another table is later created with, or renamed to, the old 
name, both tables share the same partition directories. Dropping the renamed 
managed table then deletes those directories, because its partition locations 
are outside its own table directory, and the other table loses its data.
h3. How to reproduce
 # Create a managed partitioned table with at least one partition.
 # Make the first {{RawStore.alterPartitions}} call fail with a 
{{{}JDODataStoreException{}}}.
 # Rename the table through the Thrift API with 
{{hive.metastore.hmshandler.retry.attempts}} > 1.

The call succeeds, the new table directory does not exist, and the partitions 
still point at the old directory. A unit test doing exactly this with 
{{InjectableBehaviourObjectStore}} is included in the patch.
h3. Proposed fix

Remember the location passed in by the caller before rewriting it, and put it 
back when the transaction does not commit, so that a retry sees the same input 
as the first attempt.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to