There is an other thing I care about: in spark 3.2.0, parquet 1.12 introduces
the function of encryption.
https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#columnar-encryption
Do we have plans to introduce this feature into hudi? I think this function is
a great help to the security of the data lake
发件人: Raymond Xu <xu.shiyan.raym...@gmail.com>
日期: 星期一, 2022年1月10日 上午5:48
收件人: dev@hudi.apache.org <dev@hudi.apache.org>
主题: [DISCUSS] Dropping Spark 3.0.x support in 0.11
Hi all,
The incompatible changes from different Spark 3 versions, namely spark
3.0.x, spark 3.1.x, and the latest spark 3.2, are making the spark support
difficult. I'm proposing to drop Spark 3.0.x support from the next major
release 0.11.0. Spark 3.0.x has been supported since 0.7.0 through 0.10.0.
The Spark 3 support matrix will look like
- 0.11.0
- 3.2.0 (default build), 3.1.x
- 0.10.x
- 3.1.x (default build), 3.0.x
- 0.7.0 - 0.9.0
- 3.0.x
- 0.6.0 and prior
- not supported
Thanks.
Best,
Raymond
