calleo opened a new issue #3713: URL: https://github.com/apache/hudi/issues/3713
**Describe the problem you faced** I am trying to get Hudi + Hive sync to run in a docker container. The end-goal is to have a docker image which can be used to run tests locally on a dev machine. To start with, I created a simple job that writes to a Hudi table and then reads it. While running this I have observed that reading fails with the error: `ERROR XSDB6: Another instance of Derby may have already booted the database /opt/bitnami/spark/metastore_db` The only way I can get it to work is by writing and reading using separate spark jobs, or start derby in a separate process and use jdbc to connect. Looks like the hive-sync functionality locks the databases, and won't release it until the process ends (stopping the spark session does not help). Is there any way to get around this? **To Reproduce** Steps to reproduce the behavior: Created a docker image and script that demonstrates the issue. 1. Clone https://github.com/calleo/hudi-spark-standalone 2. Run `make run_fail_1` 3. Run `make run_fail_2` **Expected behavior** It should be possible to read from a Hudi table, that has been previously created by the same Spark job. **Environment Description** * Hudi version : 0.9 * Spark version : 3.1.1 * Hive version : * Hadoop version : * Storage (HDFS/S3/GCS..) : Local * Running on Docker? (yes/no) : yes **Additional context** **Stacktrace** ``` Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /opt/bitnami/spark/metastore_db. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.raw.RawStore$6.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.RawStore.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager$5.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.access.RAMAccessManager.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.db.BasicDatabase$5.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.db.BasicDatabase.bootServiceModule(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(Unknown Source) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
