luoyuxia commented on code in PR #19573:
URL: https://github.com/apache/flink/pull/19573#discussion_r859313505
##########
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HivePartitionFetcherContextBase.java:
##########
@@ -139,16 +138,9 @@ public List<ComparablePartitionValue>
getComparablePartitionValueList() throws E
tablePath.getDatabaseName(),
tablePath.getObjectName(),
Short.MAX_VALUE);
- List<String> newNames =
- partitionNames.stream()
- .filter(
- n ->
-
!partValuesToCreateTime.containsKey(
-
extractPartitionValues(n)))
- .collect(Collectors.toList());
List<Partition> newPartitions =
metaStoreClient.getPartitionsByNames(
- tablePath.getDatabaseName(),
tablePath.getObjectName(), newNames);
+ tablePath.getDatabaseName(),
tablePath.getObjectName(), partitionNames);
Review Comment:
Then it will always contain the all paritions, finally return all partitions
with create time, right?
I prefer to only return the partitions that dose changes. Maybe you can
compare the modifcation time for all partitions to decide which partition is
updated and should be returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
