[
https://issues.apache.org/jira/browse/IMPALA-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729070#comment-16729070
]
ASF subversion and git services commented on IMPALA-7047:
---------------------------------------------------------
Commit c59f76186aa98c1bd418d2d2951561f0521edec8 in impala's branch
refs/heads/master from Todd Lipcon
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c59f761 ]
IMPALA-7047. Refreshing partitions should not make an RPC per file
The code to handle REFRESH of a single partition was incorrectly
ignoring the previously-known file descriptors. This meant that, instead
of only calling 'getFileBlockLocations' on the files that had changed
since the prior load, it was instead calling it on every file.
In addition to refresh of single partitions this also affected refresh
of unpartitioned tables (which is implemented as a refresh of its single
"default" partition).
This patch fixes the behavior by copying over the existing file
descriptor list into the re-created partition before refreshing it. A
new unit test uses FS statistics to verify the change. The new
assertions act as a regression test and fail if I comment out the fix.
Additionally, this fixes the case where the old partition had no files
to use the optimized 'listLocatedStatus' call. This is triggered when
'REFRESH' picks up a new partition from the HMS added by an external
system.
I also tested this by pointing my dev box at a remote filesystem that
was approximately 60ms away. The initial load of an unpartitioned table
with approximately 45000 files takes around 23 seconds in this setup.
Without the patch in place, REFRESH was taking upwards of 35 minutes (I
got tired and gave up at this point). Multiplying the 60ms round trip by
45000 files estimates 45 minutes. With the fix in place, REFRESH of the
same table took around 4.5 seconds.
Clearly, in typical setups where catalogd and HDFS are on a shared local
network, the gains won't be so dramatic. But, even with a 1ms round trip
(plausible when including fixed RPC overhead and potentially congested
datacenter networks) this would save 45 seconds on this example table
with 45000 files.
Change-Id: I2051b96599206164aaa06ecbdf64374c46eda956
Reviewed-on: http://gerrit.cloudera.org:8080/11227
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> REFRESH on unpartitioned tables calls getBlockLocations on every file
> ---------------------------------------------------------------------
>
> Key: IMPALA-7047
> URL: https://issues.apache.org/jira/browse/IMPALA-7047
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.13.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Labels: metadata
>
> In HdfsTable.updateUnpartitionedTableFileMd() the existing default Partition
> object is reset, and a new empty one is created. It then calls
> refreshPartitionFileMetadata with this new partition which has an empty list
> of file descriptors. This ends up listing the directory, and for each file,
> since it doesn't find it in the empty descriptor list, will make a separate
> RPC to HDFS to get the locations.
> This is quite wasteful vs just using the API that returns the located
> statuses for the directory.
> Alternatively, it seems like it should probably keep around the old file
> descriptor list in the new Partition object so that the incremental refresh
> path can work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
