Thank you for the correction, Yikun. Yes, it's 3.3.1. :) On 2021/06/17 09:03:55, Yikun Jiang <yikunk...@gmail.com> wrote: > - Apache Hadoop 3.3.2 becomes the default Hadoop profile for Apache Spark > 3.2 via SPARK-29250 today. We are observing big improvements in S3 use > cases. Please try it and share your experience. > > It should be Apache Hadoop 3.3.1 [1]. : ) > > Note that Apache hadoop 3.3.0 is the first Hadoop release including x86 and > aarch64, and 3.3.1 also. Very happy to see 3.3.1 can be the default > dependency of Spark 3.2.0. > > [1] https://hadoop.apache.org/release/3.3.1.html > > Regards, > Yikun > > > Dongjoon Hyun <dongj...@apache.org> 于2021年6月17日周四 上午5:58写道： > > > This is a continuation of the previous thread, `Apache Spark 3.2 > > Expectation`, in order to give you updates. > > > > - > > https://lists.apache.org/thread.html/r61897da071729913bf586ddd769311ce8b5b068e7156c352b51f7a33%40%3Cdev.spark.apache.org%3E > > > > First of all, the AS-IS schedule is here > > > > - https://spark.apache.org/versioning-policy.html > > > > July 1st Code freeze. Release branch cut. > > Mid July QA period. Focus on bug fixes, tests, stability and docs. > > Generally, no new features merged. > > August Release candidates (RC), voting, etc. until final release passes > > > > Second, Gengliang Wang volunteered as a release manager and started to > > work as a release manager. Thank you! He shared the on-going issues and I > > want to piggy-back the followings to his list. > > > > > > # Languages > > > > - Scala 2.13 Support: Although SPARK-25075 is almost done and we have > > Scala 2.13 Jenkins job on master branch, we do not support Scala 2.13.6. We > > should document it if Scala 2.13.7 is not arrived on time. > > Please see https://github.com/scala/scala/pull/9641 (Milestone Scala > > 2.13.7). > > > > - SparkR CRAN publishing: Apache SparkR 3.1.2 is in CRAN as of today, but > > we get policy violation warnings for cache directory. The fix deadline is > > 2021-06-28. If that's going to be removed again, we need to retry via > > Apache Spark 3.2.0 after making some fix. > > https://cran.r-project.org/web/packages/SparkR/index.html > > > > > > # Dependencies > > > > - Apache Hadoop 3.3.2 becomes the default Hadoop profile for Apache Spark > > 3.2 via SPARK-29250 today. We are observing big improvements in S3 use > > cases. Please try it and share your experience. > > > > - Apache Hive 2.3.9 becomes the built-in Hive library with more HMS > > compatibility fixes recently. We need re-evaluate the previous HMS > > incompatibility reports. > > > > - K8s 1.21 is released May 12th. K8s Client 5.4.1 supports it in Apache > > Spark 3.2. In addition, public cloud vendors start to support K8s 1.20. > > Please note that this is a breaking K8s API change from K8s Client 4.x to > > 5.x. > > > > - SPARK-33913 upgraded Apache Kafka Client dependency to 2.8.0 and Kafka > > community is considering the deprecation of Scala 2.12 support at Apache > > Kafka 3.0. > > > > - SPARK-34542 upgraded Apache Parquet dependency to 1.12.0. However, we > > need SPARK-34859 to fix column index issue before release. In addition, > > Apache Parquet encryption is added as a developer API. Custom KMS client > > should be implemented. > > > > - SPARK-35489 upgraded Apache ORC dependency to 1.6.8. We still need > > ORC-804 for better masking feature additionally. > > > > - SPARK-34651 improved ZStandard support with ZStandard 1.4.9 and we are > > currently evaluating newly arrived ZStandard 1.5.0 additionally. Currently, > > JDK11 performance is under investigation. In addition, SPARK-35181 (Use > > zstd for spark.io.compression.codec by default) is still on the way > > seperately. > > > > > > # Newly arrived items > > > > - SPARK-35779 Dynamic filtering for Data Source V2 > > > > - SPARK-35781 Support Spark on Apple Silicon on macOS natively > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > >
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
