diqiu50 commented on PR #12414: URL: https://github.com/apache/gravitino/pull/12414#issuecomment-5263336914
I think we can simplify the layout further. First, we don't need to keep Spark 3.3 support, so there is no need to introduce a `spark33` source set. Second, I don't think `spark-common` needs to remain a Gradle module. I would prefer to make it a shared source set that is compiled directly by each supported Spark version. This avoids giving `spark-common` an implicit Spark/JDK/Scala version and keeps each version fully independent. Conceptually: ```text 3.4 -> common + spark34 3.5 -> common + spark35 4.0 -> common + spark40 ``` The tests should follow the same model: shared test sources should be compiled separately for each Spark version rather than distributed as a precompiled `testArtifacts` JAR. The Glue download task and similar build logic can be moved out separately. I don't think build-time utilities are a strong enough reason to keep `spark-common` as a standalone module. I'm also fine with a reasonable amount of code duplication between version-specific implementations. Keeping clear version boundaries, avoiding cross-version dependencies, and making each supported Spark version independently removable are more important than eliminating every duplicated line of code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
