advancedxy commented on PR #939: URL: https://github.com/apache/incubator-uniffle/pull/939#issuecomment-1608617724
> > Could we shade jersey and use the shaded jersey? > > Yes, but i feel a little trouble. Currently, it just not compatible with hadoop when running UTs. Should we create a new repository for third-part libraries? Just like https://github.com/apache/hbase-thirdparty. Do you have any other good suggestions? @advancedxy @zuston @jerqi I did some test last night, it seems like a tough problem and we really don't have any simple and elegant solution. The best option (configurable and IDEA friendly) we can get is to create our own third-party repo and publish shaded jars like how hbase did it. However Uniffle itself currently do source release only, it would take extra steps to set new repo and release shaded jar to maven central. So the first baby step would be that let's try to use `hbase-thirdparty` first. @yl09099 would you mind to check whether your pr #963 could be adapted with hbase-thirdparty-jersery and is it too much trouble? After that, there might be several further options: 1. after Uniffle itself publishing binary artifact(should be done in the next release for gluten integration), we can start to take care of the Uniffle third-party repo and migrate to the new artifact. 2. Consider migrate Maven to Gradle when https://youtrack.jetbrains.com/issue/IDEA-321592 is fixed, this is a destructive change and shall require a developer(including committers and contributors) consensus vote. 3. Isolated MiniDFSCluster in a separated class loader, similar as how Spark does for its org.apache.spark.sql.hive.client.IsolatedClientLoader hive clients. P.S: I tried this option last night, it didn't succeeded but would be possible, however it would require a lot engineering effort just for test purpose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
