pyh8023 opened a new issue, #10508:
URL: https://github.com/apache/gravitino/issues/10508
### Version
main branch
### Describe what's wrong
In the project for testing the Optimizer, when submitting the built-in
update stats job, the following error occurs. The task was submitted
successfully but failed during execution. Upon checking the task error log, a
NoClassDefFoundError for the BasicDataSource class was reported, even though
this class exists in the gravitino-jobs-1.2.0.jar file.
curl -sS -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-d '{
"jobTemplateName": "builtin-iceberg-update-stats",
"jobConf": {
"catalog_name": "iceberg_test",
"table_identifier": "my_db.user_events",
"update_mode": "all",
"updater_options":
"{\"gravitino_uri\":\"http://host-196-105-0-29:8090\",\"metalake\":\"metalake\",\"statistics_updater\":\"gravitino-statistics-updater\",\"metrics_updater\":\"gravitino-metrics-updater\"}";,
"spark_conf":
"{\"spark.master\":\"local[*]\",\"spark.hadoop.fs.defaultFS\":\"hdfs://nameservice\"}",
"spark_master": "local[*]",
"spark_executor_instances": "1",
"spark_executor_cores": "1",
"spark_executor_memory": "1g",
"spark_driver_memory": "1g",
"catalog_type": "hive",
"catalog_uri":
"thrift://host-196-105-0-44:9083,thrift://host-196-105-0-49:9083",
"warehouse_location": "hdfs://nameservice/iceberg/warehouse"
}
}' \
http://host-196-105-0-29:8090/api/metalakes/metalake/jobs/runs
### Error message and/or stacktrace
parkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/gravitino/shaded/org/apache/commons/dbcp2/BasicDataSource
at
org.apache.gravitino.utils.jdbc.JdbcDataSourceFactory.create(JdbcDataSourceFactory.java:49)
at
org.apache.gravitino.maintenance.optimizer.updater.metrics.storage.jdbc.DataSourceJdbcConnectionProvider.<init>(DataSourceJdbcConnectionProvider.java:48)
at
org.apache.gravitino.maintenance.optimizer.updater.metrics.storage.jdbc.GenericJdbcMetricsRepository.initialize(GenericJdbcMetricsRepository.java:59)
at
org.apache.gravitino.maintenance.optimizer.updater.metrics.GravitinoMetricsUpdater.initialize(GravitinoMetricsUpdater.java:49)
at
org.apache.gravitino.maintenance.jobs.iceberg.IcebergUpdateStatsAndMetricsJob.createMetricsUpdater(IcebergUpdateStatsAndMetricsJob.java:460)
at
org.apache.gravitino.maintenance.jobs.iceberg.IcebergUpdateStatsAndMetricsJob.main(IcebergUpdateStatsAndMetricsJob.java:127)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:75)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.gravitino.shaded.org.apache.commons.dbcp2.BasicDataSource
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
... 18 more
26/03/19 17:00:49 INFO ShutdownHookManager: Shutdown hook called
### How to reproduce
Submit a built-in Iceberg update stats job where the catalog_type is set to
Hive.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
