Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18083
to look at the new patch set (#5).
Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing
AlterPartition event from event processor
......................................................................
[WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition
event from event processor
HdfsPartition in catalogD is a collection of files and each file is represented
by a FileDescriptor.
When the event processor processes an AlterPartitionEvent, currently it reloads
partition metadata along
with its file metadata. Reloading of file metadata is a relatively expensive
operation as it involves
listing files in the underlying filesystem.
This patch makes an improvement in processing alter partition event for
external tables. The proposed
solution is to compare HMS partition object received from AlterPartition event
and compare its storage
descriptor with the storage descriptor of already cached HdfsPartition. Reload
file metadata if any of the
following is true
* HdfsParition's sd does not match sd of HMS partition object
* Cache directive is present for HdfsPartition. File metadata reloading is done
to fetch latest
file blocks info.
Testing:
* Added new unit tests
Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2
---
A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoadOpts.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
5 files changed, 243 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/5
--
To view, visit http://gerrit.cloudera.org:8080/18083
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2
Gerrit-Change-Number: 18083
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
