prashantwason commented on code in PR #17775:
URL: https://github.com/apache/hudi/pull/17775#discussion_r2730609262
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -437,6 +439,55 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
private def arePartitionPathsUrlEncoded: Boolean =
metaClient.getTableConfig.getUrlEncodePartitioning.toBoolean
+ /**
+ * List partition paths matching path prefixes from the Catalog.
+ *
+ * File Index implementations can override this method to fetch partition
paths from the Catalog. This may be faster
+ * than listing all partition paths from the table metadata and filtering
them. This is definitely faster than
+ * listing all partition paths from the file system when metadata table may
not be enabled.
+ *
+ * Fetches all partition paths that are the sub-directories of the list of
provided (relative) paths.
+ * <p>
+ * E.g., Table has partition 4 partitions:
+ * year=2022/month=08/day=30, year=2022/month=08/day=31,
year=2022/month=07/day=03, year=2022/month=07/day=04
+ * The relative path "year=2022" returns all partitions, while the relative
path
+ * "year=2022/month=07" returns only two partitions.
+ *
+ * @param relativePathPrefixes The prefixes to relative partition paths that
must match
+ * @return null if not supported by File Index implementation
+ */
+ override protected def
getMatchingPartitionPathsFromCatalog(relativePathPrefixes: List[String]):
List[String] = {
+ // If listing from the catalog is disabled, return null
+ if
(!configProperties.getBoolean(FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.key,
FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED.defaultValue())) {
+ null
+ } else {
+ // Retrieve all the partition paths from the catalog
Review Comment:
Good point! I've updated the logic to also check if MDT is available. Now
the HMS listing path is only used when:
1. The config `FILE_INDEX_LIST_PARTITION_PATHS_FROM_HMS_ENABLED` is
explicitly enabled, AND
2. MDT is NOT available (`isMetadataTableAvailable` returns false)
This ensures we prefer the faster MDT-based listing when it's available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
