Let's keep testing 3.5.0 of course while that change is going in. (See https://github.com/apache/spark/pull/42364#issuecomment-1666878287 )
Otherwise testing is pretty much as usual, except I get this test failure in Connect, which is new. Anyone else? this is Java 8, Scala 2.13, Debian 12. - from_protobuf_messageClassName_options *** FAILED *** org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_PROTOBUF_CLASS] Could not load Protobuf class with name org.apache.spark.connect.proto.StorageLevel. org.apache.spark.connect.proto.StorageLevel does not extend shaded Protobuf Message class org.sparkproject.spark_protobuf.protobuf.Message. The jar with Protobuf classes needs to be shaded (com.google.protobuf.* --> org.sparkproject.spark_protobuf.protobuf.*). at org.apache.spark.sql.errors.QueryCompilationErrors$.protobufClassLoadError(QueryCompilationErrors.scala:3554) at org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptorFromJavaClass(ProtobufUtils.scala:198) at org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:156) at org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor$lzycompute(ProtobufDataToCatalyst.scala:58) at org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor(ProtobufDataToCatalyst.scala:57) at org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType$lzycompute(ProtobufDataToCatalyst.scala:43) at org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType(ProtobufDataToCatalyst.scala:42) at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:194) at org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:73) at scala.collection.immutable.List.map(List.scala:246) On Sat, Aug 5, 2023 at 5:42 PM Sean Owen <sro...@gmail.com> wrote: > I'm still testing other combinations, but it looks like tests fail on Java > 17 after building with Java 8, which should be a normal supported > configuration. > This is described at https://github.com/apache/spark/pull/41943 and looks > like it is resolved by moving back to Scala 2.13.8 for now. > Unless I'm missing something we need to fix this for 3.5 or it's not clear > the build will run on Java 17. > > On Fri, Aug 4, 2023 at 5:45 PM Yuanjian Li <xyliyuanj...@gmail.com> wrote: > >> Please vote on releasing the following candidate(RC1) as Apache Spark >> version 3.5.0. >> >> The vote is open until 11:59pm Pacific time Aug 9th and passes if a >> majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >> >> [ ] +1 Release this package as Apache Spark 3.5.0 >> >> [ ] -1 Do not release this package because ... >> >> To learn more about Apache Spark, please see http://spark.apache.org/ >> >> The tag to be voted on is v3.5.0-rc1 (commit >> 7e862c01fc9a1d3b47764df8b6a4b5c4cafb0807): >> >> https://github.com/apache/spark/tree/v3.5.0-rc1 >> >> The release files, including signatures, digests, etc. can be found at: >> >> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-bin/ >> >> Signatures used for Spark RCs can be found in this file: >> >> https://dist.apache.org/repos/dist/dev/spark/KEYS >> >> The staging repository for this release can be found at: >> >> https://repository.apache.org/content/repositories/orgapachespark-1444 >> >> The documentation corresponding to this release can be found at: >> >> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-docs/ >> >> The list of bug fixes going into 3.5.0 can be found at the following URL: >> >> https://issues.apache.org/jira/projects/SPARK/versions/12352848 >> >> This release is using the release script of the tag v3.5.0-rc1. >> >> >> FAQ >> >> ========================= >> >> How can I help test this release? >> >> ========================= >> >> If you are a Spark user, you can help us test this release by taking >> >> an existing Spark workload and running on this release candidate, then >> >> reporting any regressions. >> >> If you're working in PySpark you can set up a virtual env and install >> >> the current RC and see if anything important breaks, in the Java/Scala >> >> you can add the staging repository to your projects resolvers and test >> >> with the RC (make sure to clean up the artifact cache before/after so >> >> you don't end up building with an out of date RC going forward). >> >> =========================================== >> >> What should happen to JIRA tickets still targeting 3.5.0? >> >> =========================================== >> >> The current list of open tickets targeted at 3.5.0 can be found at: >> >> https://issues.apache.org/jira/projects/SPARK and search for "Target >> Version/s" = 3.5.0 >> >> Committers should look at those and triage. Extremely important bug >> >> fixes, documentation, and API tweaks that impact compatibility should >> >> be worked on immediately. Everything else please retarget to an >> >> appropriate release. >> >> ================== >> >> But my bug isn't fixed? >> >> ================== >> >> In order to make timely releases, we will typically not hold the >> >> release unless the bug in question is a regression from the previous >> >> release. That being said, if there is something which is a regression >> >> that has not been correctly targeted please ping me or a committer to >> >> help target the issue. >> >> Thanks, >> >> Yuanjian Li >> >>
