rdblue commented on pull request #3237: URL: https://github.com/apache/iceberg/pull/3237#issuecomment-938956754
Thanks for getting this started, @jackye1995! I think this is a good first step, but there are a few things we should consider. First, I agree that we probably want to parameterize the Scala version while we're doing this work. We are going to need to address it sooner or later and now seems like a good time to at least put in a parameter that is always 2.12 so that we can get a 2.13 build working later. Next, I think we need to think about package naming. Right now, we produce `iceberg-spark-runtime`, `iceberg-spark3-runtime`, etc. I think that this produces just the `iceberg-spark-runtime` and not the `spark3` Jars, which would possibly break scripts because people depend on certain module names. We will also need to publish multiple Spark Jars so I think we need to give them unique names. Scala has the convention of adding the Scala version into the module name like `_2.12`. I think we should probably do the same and start publishing `iceberg-spark-runtime_3.2_2.12` and `iceberg-spark_3.2_2.12`. We can continue to publish Spark 2.4 as `iceberg-spark-runtime` and Spark 3.0 as `iceberg-spark3-runtime` for consistency. Both of those have fixed Scala versions anyway so that's good. Since we probably want to be able to publish Jars for all Spark versions, I think it makes sense to be able to build the whole project with every Spark version. And since we will probably be embedding version strings in module names, I'm wondering if the conditional project logic in the main `build.gradle` is a good idea anymore. Gradle now recommends a separate `build.gradle` for each subproject, which would work well. We could have a `build.gradle` specific to each `spark/v$major.$minor` build directory and select whole projects using `settings.gradle`. That avoids big changes to `build.gradle` and makes it easy to evolve each Spark build independently. I went ahead and tried out updating this build like this and the result is #3256. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
