[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12 URL: https://github.com/apache/incubator-hudi/pull/1109#issuecomment-574918036 > @ezhux : I see a bunch of earlier comments. Some of these are addressed in PR #1226 . Do you think you will be able to address the changes in this PR in a day so that we can land it by friday (Sunday is the release cutoff date). If not, we can focus on landing PR #1226 ( cc @ezhux ) > > @zhedoubushishi : Thanks for opening another diff. > > @ezhux @zhedoubushishi : After spending sometime playing with maven to pull off cross-compiling, i think it is easier to go with profile approach (e:g : in spark -2.4.4 setup - https://github.com/apache/spark/blob/branch-2.4/external/avro/pom.xml and profile definition in base pom : https://github.com/apache/spark/blob/branch-2.4/pom.xml#L2797 > > The main change we need would to rename hudi-spark to hudi-spark_2.11 for 2.11 and hudi-spark_2.12 (when run with overridden profile. Same for hudi-spark-bundle and hudi-utilities-bundle ? > > If this works, we can change the release process to generate jars for both 2.11 and 2.12. Let me know your thoughts. I am trying to use the Spark way to support Scala 2.12. (https://spark.apache.org/docs/latest/building-spark.html) To build Hudi with Scala 2.12, first change the major Scala version using (e.g. 2.12): ``` dev/change-scala-version.sh 2.12 ``` Then build with Maven: ``` mvn -Pscala-2.12 clean install ``` Is this the way you want? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12 URL: https://github.com/apache/incubator-hudi/pull/1109#issuecomment-574886880 > > @bvaradar : I reviewed #1226 and see that it does the same as my PR, but is cleaner (I have force-pushed multiple times). So if #1226 works, I'm fine with closing this one. > > Sounds good @ezhux Thanks a lot for this PR though. It helped us see how to approach this issue. > > @zhedoubushishi : Would you be able to address the comments ([#1109 (review)](https://github.com/apache/incubator-hudi/pull/1109#pullrequestreview-343402313)) in #1226 to get ready for landing. Sure I can take it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12 URL: https://github.com/apache/incubator-hudi/pull/1109#issuecomment-574321102 > > Sure. I will send another PR. Currently our work only supports 2.12, but I can try to see if it is possible to support both 2.11 and 2.12. > > @zhedoubushishi : Is your change different from what is being done as part of this PR ? Anyways, it would help if you can open a WIP PR and we can cross check with this PR to see if we are missing anything here. > > Also @zhedoubushishi @ezhux : I see this info in stack-overflow to build both 2.11 and 2.12 versions of packages. https://stackoverflow.com/a/46785150. Can you check if this model would work for hudi ? We would need to change pom for hudi-spark and its dependents : hudi-spark-bundle and hudi-utilities-bundle I created a PR here: https://github.com/apache/incubator-hudi/pull/1226. This PR is compatible with Scala 2.12 so you can build it with following command: ``` mvn clean install -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -DskipTests ``` I am not sure if this https://stackoverflow.com/a/46785150 could work for Hudi. My understanding is Hudi has many Scala dependencies. Say by default, scala.version=2.11, then hudi-spark will depend on many xxx_2.11 packages. Then does it make sense to compile Hudi code with both Scala 2.12 and Scala 2.11? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12 URL: https://github.com/apache/incubator-hudi/pull/1109#issuecomment-573847670 > @zhedoubushishi : As you had mentioned that AWS EMR has internally made it possible to package hudi jars using scala 2.12, can you shepherd this PR ? This is one of the critical PRs to be fixed before next week (deadline end of week). > > I also have a question here : Has AWS EMR migrated the scala compile version to 2.12 or are you supporting both 2.11 and 2.12 ? It looks like spark-2.4.4 (which is used for compiling Hudi) has both 2.11 and 2.12 packaging support. So, wondering if we can support both 2.11 and 2.12 hudi package generation. Let us know. Sure. I will send another PR. Currently our work only supports 2.12, but I can try to see if it is possible to support both 2.11 and 2.12. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
zhedoubushishi commented on issue #1109: [HUDI-238] - Migrating to Scala 2.12
URL: https://github.com/apache/incubator-hudi/pull/1109#issuecomment-566914616
> hi @zhedoubushishi ,
>
> 1. I didn't know that I can import it through maven, would have saved me
quite some time... Will do this
> 2. thanks for pointing this out, didn't know either
> 3. thanks, will have a look.
I guess you can try to import it like this:
```
org.apache.spark
spark-streaming-kafka-0-10_2.12
${spark.version}
tests
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
