jessiedanwang opened a new issue, #5579:
URL: https://github.com/apache/iceberg/issues/5579
### Apache Iceberg version
0.14.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Hi, there,
I have tried to register udf, see below,
val bucketF = "bucketn"
IcebergSpark.registerBucketUDF(spark, bucketF, DataTypes.LongType,
numBuckets)
if(spark.sql(s"show USER functions $bucketF").count == 0) {
log.error("unable to register bucket function") //it is not in the
log, so looks like bucketn udf is registered
}
However, the following line of code throw errors,
df.sortWithinPartitions(expr(s"bucketn($partitionKey)"))
22/08/18 22:02:10 ERROR GlueMetastoreClientDelegate:
com.amazonaws.services.glue.model.EntityNotFoundException: Cannot find
function. (Service: AWSGlue; Status Code: 400; Error Code:
EntityNotFoundException; Request ID: 9021d50b-5fe9-421f-8bcd-20a67cd73b5f;
Proxy: null)
22/08/18 22:02:10 ERROR MicroBatchExecution: Query iceberg/cdc/xxx [id =
ea711eec-c2c5-40f2-a562-b07db4851429, runId =
617c24bb-e891-4853-b2b4-05aaf0e31b2b] terminated with error
org.apache.spark.sql.AnalysisException: Undefined function: 'bucketn'. This
function is neither a registered temporary function nor a permanent function
registered in the database 'default'
The log "unable to register bucket function" does not appear in the log, so
looks liek that bucketn udf is registered, However, the above Glue error seems
to indicate that the udf can not be found in Glue? I have tried the following
glue command on EMR 6.5, and it returns nothing,
aws glue get-user-defined-functions --catalog-id 300772411908 --pattern
bucketn
{
"UserDefinedFunctions": []
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
