LuciferYang opened a new issue, #11010:
URL: https://github.com/apache/paimon/issues/11010

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (c2e001b7f2)
   
   ### Compute Engine
   
   Spark
   
   ### Minimal reproduce step
   
   `SparkGenericCatalogWithHiveTest#testBuildWithHive` and 
`#testAddPartitionForFallbackParquetTableWithNamedCatalog` fail in the `Java / 
Spark 4 / Scala 2.13` CI job when a test class that creates a `HiveCatalog` 
with `hive-conf-dir` (for example `SparkCatalogWithHiveTest`) ran earlier in 
the same surefire fork. Recent occurrences: 
[#9265](https://github.com/apache/paimon/actions/runs/41961353543/job/127030997038),
 [another 
PR](https://github.com/apache/paimon/actions/runs/41956536661/job/127013907079).
   
   ```
   Exception in thread "TThreadPoolServer WorkerProcess-2" 
java.lang.NoClassDefFoundError: com/jolbox/bonecp/BoneCPConfig
        at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.setupJdbcConnectionPool(TxnHandler.java:3156)
        at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.setConf(TxnHandler.java:260)
        at 
org.apache.hadoop.hive.metastore.txn.TxnUtils.getTxnStore(TxnUtils.java:109)
        at 
org.apache.hadoop.hive.metastore.AcidEventListener.getTxnHandler(AcidEventListener.java:84)
        at 
org.apache.hadoop.hive.metastore.AcidEventListener.onDropDatabase(AcidEventListener.java:50)
        ...
   [ERROR] 
org.apache.paimon.spark.SparkGenericCatalogWithHiveTest.testBuildWithHive(Path) 
 <<< ERROR!
   org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.metastore.api.NoSuchObjectException: There is no 
database named my_db
   ```
   
   ### What doesn't meet your expectations?
   
   The test metastore should not depend on what an earlier test class in the 
same JVM did.
   
   `HiveCatalog.createHiveConf` with `hive-conf-dir` calls 
`HiveConf.setHiveSiteLocation(null)`, `setLoadMetastoreConfig(false)` and 
`setLoadHiveServer2Config(false)`. These are JVM-wide statics and are not 
restored. A `TestHiveMetastore` started afterwards no longer reads the test 
`hive-site.xml`, so it loses `datanucleus.connectionPoolingType=DBCP` and falls 
back to Hive's default, BoneCP. BoneCP is not on the Spark 4 test classpath. On 
`DROP DATABASE` the metastore's `AcidEventListener` sets up the `TxnHandler` 
connection pool and the worker thread dies with `NoClassDefFoundError`. The 
database is already dropped at that point, so the client's retry reports it as 
missing.
   
   The Spark 3 modules have BoneCP on their test classpath, so the same order 
passes there.
   
   ### Anything else?
   
   I haven't found an effect outside tests. Spark's built-in Hive client adds 
`hive-site.xml` to its Hadoop configuration itself, so it doesn't depend on 
these statics.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to