Quanlong Huang created IMPALA-12523:
---------------------------------------
Summary: Avoid loading all partitions to get the file formats
Key: IMPALA-12523
URL: https://issues.apache.org/jira/browse/IMPALA-12523
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
There is a TODO in LocalFsTable that we should address:
{code}
public Set<HdfsFileFormat> getFileFormats() {
// TODO(todd): can we avoid loading all partitions here? this is called
// for any INSERT query, even if the partition is specified.
Collection<? extends FeFsPartition> parts;
if (ref_ != null) {
parts = FeCatalogUtils.loadAllPartitions(this);
} else {
// If this is a CTAS target, we don't want to try to load the partition
list.
parts = Collections.emptyList();
}
// In the case that we have no partitions added to the table yet, it's
// important to add the "prototype" partition as a fallback.
Iterable<FeFsPartition> partitionsToConsider = Iterables.concat(
parts, Collections.singleton(createPrototypePartition()));
return FeCatalogUtils.getFileFormats(partitionsToConsider);
}
{code}
We can consider passing the set of file formats to coordinators (local-catalog
mode) when loading the TableMetaRef, just like how we load the sql_constraints,
etc.
* Add a field of 'want_file_formats' in TTableInfoSelector.
* Catalogd add the set of file formats in the response.
* Add the set of file formats to TableMetaRefImpl and use it in
LocalFsTable.getFileFormats()
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
