bowenliang123 commented on PR #7153:
URL: https://github.com/apache/iceberg/pull/7153#issuecomment-1483977664
Thanks to @RussellSpitzer , and reviews from @pan3793 and @zhongyujiang .
And here is some more clarification to the `load Iceberg built-in function
in BaseCatalog` feature listed in the description.
Facts
1. it is not able to use Iceberg's built-in function is not working (like
`SELECT * FROM system.years(SELECT system.years(date('1970-01-01')))`), BEFORE
or AFTER this PR
2. this PR does load functions from Iceberg's SparkFunctions via BaseCatalog
as Function Catalog (confirmed in debugging), but it doesn't reolve the problem
in 1.
3. the problem comes with Spark's Analyzer, which forces to do
`resolveV1Function` with `v1SessionCatalog` instance (but NOT the v2 session
catalog !) when using the `spark_catalog` session catalog. More importantly,
the `v1SessionCatalog` used is initialized in Spark's CatalogManager inside,
and it seems not able to inject or register funcions from Iceberg's
SparkSessionCatalog CatalogPlugin.
https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#LL2118C36-L2118C36
```
case u @ UnresolvedFunction(nameParts, arguments, _, _, _) =>
withPosition(u) {
resolveBuiltinOrTempFunction(nameParts, arguments, Some(u)).getOrElse {
val CatalogAndIdentifier(catalog, ident) = expandIdentifier(nameParts)
if (CatalogV2Util.isSessionCatalog(catalog)) {
resolveV1Function(ident.asFunctionIdentifier, arguments, u)
} else {
resolveV2Function(catalog.asFunctionCatalog, ident, arguments, u)
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
