kbendick commented on code in PR #5305:
URL: https://github.com/apache/iceberg/pull/5305#discussion_r926996627
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/BaseCatalog.java:
##########
@@ -46,4 +54,56 @@ public Procedure loadProcedure(Identifier ident) throws
NoSuchProcedureException
throw new NoSuchProcedureException(ident);
}
+
+ @Override
+ public Identifier[] listFunctions(String[] namespace) throws
NoSuchNamespaceException {
+ if (isValidNamespaceForFunction(namespace)) {
+ return Arrays.stream(SparkFunctions.list())
+ .map(name -> Identifier.of(namespace, name))
+ .toArray(Identifier[]::new);
+ } else {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ @Override
+ public UnboundFunction loadFunction(Identifier ident) throws
NoSuchFunctionException {
+ String[] namespace = ident.namespace();
+ String name = ident.name();
+
+ if (isValidNamespaceForFunction(namespace)) {
+ UnboundFunction func = SparkFunctions.load(name);
+ if (func != null) {
+ return func;
+ }
+ }
+
+ throw new NoSuchFunctionException(ident);
+ }
+
+ /**
+ * When in an Iceberg catalog, allow for using the built-in Iceberg
functions provided that:
+ * <ol>
+ * <li>
+ * The namespace is not specified,
+ * i.e. {@code SELECT truncate(1, 4)} or {@code SELECT
my_catalog.truncate(1, 4)}
+ * </li>
+ * <li>
+ * The implicit <b>system</b> namespace is used, to mirror call
procedure syntax,
+ * i.e. {@code SELECT system.truncate(1, 4)} or {@code SELECT
my_catalog.system.truncate(1, 4)}
+ * </li>
+ * <li>
+ * A namespace that exists within the catalog is referenced,
+ * i.e. {@code SELECT ns.truncate(1, 4)} or {@code SELECT
my_catalog.ns.truncate(1, 4)}
Review Comment:
So the only problem with this is that within SQL (or even the dataframe
API), the `SparkSessionCatalog` checks that the namespace exists.
So calling `system.truncate` results in:
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/BaseCatalog.java:
##########
@@ -46,4 +54,56 @@ public Procedure loadProcedure(Identifier ident) throws
NoSuchProcedureException
throw new NoSuchProcedureException(ident);
}
+
+ @Override
+ public Identifier[] listFunctions(String[] namespace) throws
NoSuchNamespaceException {
+ if (isValidNamespaceForFunction(namespace)) {
+ return Arrays.stream(SparkFunctions.list())
+ .map(name -> Identifier.of(namespace, name))
+ .toArray(Identifier[]::new);
+ } else {
+ throw new NoSuchNamespaceException(namespace);
+ }
+ }
+
+ @Override
+ public UnboundFunction loadFunction(Identifier ident) throws
NoSuchFunctionException {
+ String[] namespace = ident.namespace();
+ String name = ident.name();
+
+ if (isValidNamespaceForFunction(namespace)) {
+ UnboundFunction func = SparkFunctions.load(name);
+ if (func != null) {
+ return func;
+ }
+ }
+
+ throw new NoSuchFunctionException(ident);
+ }
+
+ /**
+ * When in an Iceberg catalog, allow for using the built-in Iceberg
functions provided that:
+ * <ol>
+ * <li>
+ * The namespace is not specified,
+ * i.e. {@code SELECT truncate(1, 4)} or {@code SELECT
my_catalog.truncate(1, 4)}
+ * </li>
+ * <li>
+ * The implicit <b>system</b> namespace is used, to mirror call
procedure syntax,
+ * i.e. {@code SELECT system.truncate(1, 4)} or {@code SELECT
my_catalog.system.truncate(1, 4)}
+ * </li>
+ * <li>
+ * A namespace that exists within the catalog is referenced,
+ * i.e. {@code SELECT ns.truncate(1, 4)} or {@code SELECT
my_catalog.ns.truncate(1, 4)}
Review Comment:
So the only problem with this is that within SQL (or even the dataframe
API), the `SparkSessionCatalog` checks that the namespace exists.
So calling `system.truncate` results in
```
org.apache.spark.sql.AnalysisException: Undefined function: 'truncate'. This
function is neither a registered temporary function nor a permanent function
registered in the database 'system'.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
