[
https://issues.apache.org/jira/browse/FLINK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated FLINK-23010:
---------------------------
Description: When consume order is {{CREATE_TIME}},
HivePartitionFetcherContextBase lists folders under table location to find
partitions. This is wrong because HMS is the single source of truth to decide
whether a partition exists for a hive table. Returning a non-existing partition
will lead to unnecessary job FO when {{HiveContinuousPartitionFetcher}} fails
to get the partition from HMS. (was: When consume order is {{CREATE_TIME}}, we
list folders under table location )
> HivePartitionFetcherContextBase::getComparablePartitionValueList can return
> partitions that don't exist
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-23010
> URL: https://issues.apache.org/jira/browse/FLINK-23010
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Reporter: Rui Li
> Assignee: Rui Li
> Priority: Major
>
> When consume order is {{CREATE_TIME}}, HivePartitionFetcherContextBase lists
> folders under table location to find partitions. This is wrong because HMS is
> the single source of truth to decide whether a partition exists for a hive
> table. Returning a non-existing partition will lead to unnecessary job FO
> when {{HiveContinuousPartitionFetcher}} fails to get the partition from HMS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
