Quanlong Huang created IMPALA-14646:
---------------------------------------
Summary: StorageDescriptor normalization should deal with
parameters
Key: IMPALA-14646
URL: https://issues.apache.org/jira/browse/IMPALA-14646
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
When checking whether an ALTER_TABLE event has trivial changes in
StorageDescriptor, we normalize fields that are unrelated to file metadata,
e.g. cols, bucketCols, sortCols, etc. See code in
[normalizeStorageDescriptor()|https://github.com/apache/impala/blob/85d77b908b12ae3d3f48ed5d49f38fb3832edc4e/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L2334-L2347].
However, the parameters map of StorageDescriptor is not normalized, which
causes null and empty map be considered as different:
{code:bash}
I1217 21:53:42.995173 279445 MetastoreEvents.java:825] EventId: 365355029
EventType: ALTER_TABLE Non-trivial change in table storage descriptor (SD)
detected for table my_db.my_tbl. So file metadata should be reloaded. SD
before: StorageDescriptor(cols:[...], location:hdfs://table_path,
inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat,
outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat,
compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}),
bucketCols:[], sortCols:[], parameters:{},
skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false), SD after:
StorageDescriptor(cols:[...], location:hdfs://table_path,
inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat,
outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat,
compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}),
bucketCols:null, sortCols:null, parameters:null){code}
In the above log, the old SD has an empty parameters map: {}, the new SD has a
null parameters map. This should be considered as trivial change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
