flaming-archer commented on code in PR #7123:
URL: https://github.com/apache/kyuubi/pull/7123#discussion_r2189559602
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/read/HiveFileIndex.scala:
##########
@@ -54,6 +54,13 @@ class HiveCatalogFileIndex(
override def partitionSchema: StructType = table.partitionSchema
+ override def listFiles(
Review Comment:
1. If this method is not overridden, it will call the `listFiles` method of
the parent class's PartitioningAvareFileIndex, which needs to override the
`pathSpec` method.
Rewriting this method is a bit difficult because it requires knowing
the specific value of the partition.
See definition `def partitionSpec(): PartitionSpec`
And the definitions of PartitionSpec are
`case class PartitionPath(values: InternalRow, path: Path)` , pls
notice the **values**.
`case class PartitionSpec(`
`partitionColumns: StructType,`
`partitions: Seq[PartitionPath])`
2. I think hiveFileidex has actually rewritten the listFiles method in
HiveFileIndex.scala, so if we reuse it directly, the risk should be lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org
For additional commands, e-mail: notifications-h...@kyuubi.apache.org
