zhouyuan commented on code in PR #12833:
URL: https://github.com/apache/gluten/pull/12833#discussion_r4120172572
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -115,6 +115,32 @@ object GlutenIcebergSourceUtil {
)
}
+ /**
+ * Returns the table-level root path(s) so callers can validate the
underlying filesystem
+ * scheme(s) without enumerating every data file.
+ *
+ * We deliberately read this from table metadata (Table.location() plus the
write.data.path /
+ * write.folder-storage.path properties, when set) rather than from the
actual planned scan tasks
+ * (task.file().path()): resolving the real per-file paths requires planning
the Iceberg scan's
+ * input partitions, but doing so eagerly -- before Spark has pushed down
its dynamic partition
+ * pruning runtime filters -- breaks DPP's subquery-reuse detection for
SupportsRuntimeV2Filtering
+ * scans (see https://github.com/apache/gluten/issues/12712).
+ *
+ * This mirrors how the non-Iceberg
BatchScanExecTransformer.getRootPathsInternal resolves root
+ * paths purely from FileIndex metadata (fileScan.fileIndex.rootPaths)
without planning any input
+ * partitions. Just like that metadata-only approach, this does not reflect
files relocated by an
+ * entirely custom LocationProvider that ignores both properties above, but
it is a strict
+ * improvement over unconditionally returning Seq.empty.
+ */
+ def getRootPaths(table: Table): Seq[String] = {
+ val properties = table.properties()
+ Seq(
+ Option(table.location()),
+ Option(properties.get(TableProperties.WRITE_DATA_LOCATION)),
+ Option(properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION))
Review Comment:
Using metadata location is a great idea, is there a API from iceberg that
returns exactly one path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
