wombatu-kun commented on code in PR #18975:
URL: https://github.com/apache/hudi/pull/18975#discussion_r4573555063
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -1338,6 +1333,45 @@ class HoodieSparkSqlWriterInternal {
}
object HoodieSparkSqlWriterInternal {
+ private val log =
LoggerFactory.getLogger(classOf[HoodieSparkSqlWriterInternal])
+
+ /**
+ * Best-effort invalidation of the Spark catalog relation cache for the
just-synced table(s), so
+ * subsequent reads in the same Spark session reflect the new write.
+ *
+ * The table name is always qualified with the sync database
([[HIVE_DATABASE]] /
+ * `hoodie.datasource.hive_sync.database`) so a same-named table in another
database (in
+ * particular the session's current/`default` database) is never resolved
and refreshed by
+ * mistake.
+ *
+ * Any failure is logged and swallowed: by this point the data has already
been committed and
+ * meta-synced successfully, so a cache-invalidation problem (a transient
catalog error, or a
+ * same-named table backed by storage the writer cannot access) must never
fail the write.
+ */
+ def refreshSparkCatalogTableCache(spark: SparkSession, syncDb: String,
tableNames: Seq[String]): Unit = {
Review Comment:
The AccessDenied in #18139 is thrown by
`HoodieFileIndex.getConfigProperties` looking up `<current
db>.<hoodie.table.name>` while the correctly qualified sync-db table is being
resolved, and on master that lookup already runs only for reads that declare a
bucket index and is best-effort. Is this meant as extra hardening on top of
that, and if so could the description stop attributing #18139 to the refresh
resolving the `default` table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]