bharath v created IMPALA-5325:
---------------------------------
Summary: Update HdfsTable#totalHdfsBytes_ and
HdfsTable#numHdfsFiles_ atomically.
Key: IMPALA-5325
URL: https://issues.apache.org/jira/browse/IMPALA-5325
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: bharath v
Fix For: Impala 2.8.0, Impala 2.9.0
In the following snippet from HdfsTable.java
{noformat}
private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
HdfsPartition partition) throws Exception {
Preconditions.checkNotNull(storageDescriptor);
Preconditions.checkNotNull(partition);
Path partDirPath = new Path(storageDescriptor.getLocation());
FileSystem fs = partDirPath.getFileSystem(CONF);
if (!fs.exists(partDirPath)) return;
numHdfsFiles_ -= partition.getNumFileDescriptors();
totalHdfsBytes_ -= partition.getSize();
Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
refreshFileMetadata(partition);
}
{noformat}
If {{refreshFileMetadata}} throws an exception before updating
{{numHdfsFiles_}} and {{totalHdfsBytes_}}, the table could get into an
inconsistent state and could possibly hit a preconditions check in the next
block metadata update.
Fix: Update {{totalHdfsBytes_}} and {{numHdfsFiles_}} in an atomic fashion.
Workaround: Run "invalidate table" to clear the full table state.
This bug also affects older versions around method {{updatePartitionFds}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
