[
https://issues.apache.org/jira/browse/IMPALA-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876071#comment-17876071
]
ASF subversion and git services commented on IMPALA-12856:
----------------------------------------------------------
Commit f18b21c1ae398bd66783e7a901c8984c186c5874 in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f18b21c1a ]
IMPALA-12856: Event processor should ignore processing partition
with empty partition values
While processing partition related events, Event Processor (EP) is
facing IllegalStateException if the partition fetched from HMS has
empty partition values. Even though this is a bug in HMS which returns
partitions with empty values, EP should ignore such partitions instead
of throwing IllegalStateException.
Note: Added a debug option 'mock_empty_partition_values' to add
malformed partition objects.
Testing:
- Manually verified the test provided in jira details in local env.
- Added unit test to return empty partition values and verify EP state.
Change-Id: Id2469930ccd74948325f1723bd8b2bd6aad02d09
Reviewed-on: http://gerrit.cloudera.org:8080/21143
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> IllegalStateException in processing RELOAD events due to malformed HMS
> Partition objects
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-12856
> URL: https://issues.apache.org/jira/browse/IMPALA-12856
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Labels: catalog-2024
>
> When processing RELOAD events on partitions, catalogd fetch the Partition
> objects from HMS. The returned Partition objects could be malformed which
> causes an IllegalStateException and stops the event-processor. This was
> observed when a partition is added and dropped in a loop.
> {noformat}
> E0229 15:19:27.945312 12668 MetastoreEventsProcessor.java:990] Unexpected
> exception received while processing event
> Java exception follows:
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at
> org.apache.impala.catalog.HdfsTable.getTypeCompatiblePartValues(HdfsTable.java:2598)
> at
> org.apache.impala.catalog.HdfsTable.reloadPartitionsFromNames(HdfsTable.java:2856)
> at
> org.apache.impala.service.CatalogOpExecutor.reloadPartitionsFromNamesIfExists(CatalogOpExecutor.java:4805)
> at
> org.apache.impala.service.CatalogOpExecutor.reloadPartitionsIfExist(CatalogOpExecutor.java:4742)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartitions(MetastoreEvents.java:1050)
> at
> org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processPartitionReload(MetastoreEvents.java:2941)
> at
> org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processTableEvent(MetastoreEvents.java:2906)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1248)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:672)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> E0229 15:19:27.963455 12668 MetastoreEventsProcessor.java:1251] Event id:
> 8697728
> Event Type: RELOAD
> Event time: 1709191166
> Database name: default
> Table name: part_tbl
> Event message: H4s{noformat}
> The failed check is asserting the number of partition columns cached in
> catalogd matches the number of partition values from the HMS object:
> {code:java}
> public List<LiteralExpr> getTypeCompatiblePartValues(List<String> values) {
> List<LiteralExpr> result = new ArrayList<>();
> List<Column> partitionColumns = getClusteringColumns();
> Preconditions.checkState(partitionColumns.size() == values.size()); //
> This failed{code}
> After adding some debug logs, I found the Partition obejct got from HMS had
> an empty values list:
> {noformat}
> I0229 16:04:04.679625 25867 HdfsTable.java:2829] HMS Partition:
> Partition(values:[], dbName:default, tableName:part_tbl,
> createTime:1709193844, lastAccessTime:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:i, type:int, comment:null)],
> location:hdf
> s://localhost:20500/test-warehouse/part_tbl/p=1,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializ
> ationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{}),
> bucketCols:[], sortCols:[], parameters:{},
> skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false), par
> ameters:{}, catName:hive, writeId:0)
> I0229 16:04:04.680133 25867 MetastoreEventsProcessor.java:1189] Time elapsed
> in processing event batch: 17.145ms
> E0229 16:04:04.680475 25867 MetastoreEventsProcessor.java:990] Unexpected
> exception received while processing event
> Java exception follows:
> java.lang.IllegalStateException: partitionColumns.size(): 1, values.size():
> 0. partCols: Column{name_=p, type_=INT, comment_=null,
> stats=ColumnStats{avgSize_=4.0, avgSerializedSize_=4.0, maxSize_=4,
> numDistinct_=-1, numNulls_=-1, numTrues=-1, numFalses=-1, lowValue=-1,
> highValue=-1}, position_=0}, partValues: {noformat}
> EventProcessor should recover in this case.
> On the other hand, returning a Partition object with empty partition value
> list seems a bug of HMS. CC [~hemanth619], [~VenuReddy]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
