zhedoubushishi commented on issue #1226: [HUDI-238] Make Hudi support Scala 2.12 URL: https://github.com/apache/incubator-hudi/pull/1226#issuecomment-575513618 > > Also there is another way to do this: I can rename artifact Id, e.g.`hudi-spark` to `hudi-spark_${scala.binary.version}`, and thus avoid using `dev/change-scala-version 2.12`. I am not sure why Spark does not choose this way. > > @zhedoubushishi : If the artifact id renaming to hudi-spark_${scala.binary.version} works, please go for it and make the change. I will take a look at this PR tomorrow morning PST again. Sorry I just found ```hudi-spark_${scala.binary.version}``` might not work. See the dependency tree of hudi-cli (hudi-cli depends on hudi-utilities, and hudi-utilities depends on hudi-spark and other spark libraries): ``` $ mvn dependency:tree -Pscala-2.12 -pl hudi-cli [INFO] ----------------------< org.apache.hudi:hudi-cli >---------------------- [INFO] Building hudi-cli 0.5.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-cli --- [INFO] org.apache.hudi:hudi-cli:jar:0.5.1-SNAPSHOT [INFO] +- org.scala-lang:scala-library:jar:2.12.10:compile [INFO] +- org.apache.hudi:hudi-hive:jar:0.5.1-SNAPSHOT:compile [INFO] | +- org.apache.hudi:hudi-hadoop-mr:jar:0.5.1-SNAPSHOT:compile [INFO] | \- com.beust:jcommander:jar:1.72:compile ... [INFO] +- org.apache.hudi:hudi-utilities_2.12:jar:0.5.1-SNAPSHOT:compile [INFO] | +- org.apache.hudi:hudi-spark_2.11:jar:0.5.1-SNAPSHOT:compile [INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1:compile [INFO] | | +- org.scala-lang:scala-reflect:jar:2.11.8:compile [INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9:compile [INFO] | +- org.apache.spark:spark-streaming_2.11:jar:2.4.4:compile [INFO] | | +- org.apache.spark:spark-core_2.11:jar:2.4.4:compile [INFO] | | | +- com.twitter:chill_2.11:jar:0.9.3:compile [INFO] | | | +- org.apache.spark:spark-launcher_2.11:jar:2.4.4:compile [INFO] | | | +- org.apache.spark:spark-kvstore_2.11:jar:2.4.4:compile [INFO] | | | +- org.apache.spark:spark-network-common_2.11:jar:2.4.4:compile [INFO] | | | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.4:compile [INFO] | | | +- org.apache.spark:spark-unsafe_2.11:jar:2.4.4:compile [INFO] | | | \- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile [INFO] | | | \- org.json4s:json4s-core_2.11:jar:3.5.3:compile [INFO] | | | +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile [INFO] | | | +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile [INFO] | | | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile [INFO] | | \- org.apache.spark:spark-tags_2.11:jar:2.4.4:compile [INFO] | +- org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:2.4.4:compile [INFO] | | \- org.apache.kafka:kafka-clients:jar:2.0.0:compile [INFO] | +- org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:tests:2.4.4:compile [INFO] | +- org.antlr:stringtemplate:jar:4.0.2:compile [INFO] | | \- org.antlr:antlr-runtime:jar:3.3:compile [INFO] | +- com.twitter:bijection-avro_2.11:jar:0.9.3:compile [INFO] | | +- com.twitter:bijection-core_2.11:jar:0.9.3:compile [INFO] | | \- org.scoverage:scalac-scoverage-runtime_2.11:jar:1.3.0:compile [INFO] | +- io.confluent:kafka-avro-serializer:jar:3.0.0:compile [INFO] | +- io.confluent:common-config:jar:3.0.0:compile [INFO] | +- io.confluent:common-utils:jar:3.0.0:compile [INFO] | | \- com.101tec:zkclient:jar:0.5:compile [INFO] | +- io.confluent:kafka-schema-registry-client:jar:3.0.0:compile [INFO] | \- org.apache.httpcomponents:httpcore:jar:4.3.2:compile ``` Although scala-2.12 profile overrides the ```hudi-utilities_${scala.binary.version}``` to ```hudi-utilities_2.12```, but when it comes to the dependency of hudi-utilities_2.12, it seems maven would only use the default ```scala.binary.version``` inside ```hudi-utilities_${scala.binary.version}``` which is 2.11. And thus all the dependencies of ```hudi-utilities_${scala.binary.version}``` goes will ```xxx_2.11```.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
