Hi dev, Since v1.6, the Spark AuthZ Plugin has been introduced as a core extension for our Spark SQL engine and other standalone Spark SQL applications.
Although authz heavily depends on private modules of Spark SQL and Spark Catalyst, we have well designed an extensible serde layer for spark cross-version compatibilities, which makes us easily support all Spark 3 releases. The main reasons for not bringing Spark 2, which is not that hard, are Scala version incompatibilities, lack of use cases and contributions. The development of Spark 4 is currently underway. It then brings critical changes such as JDK and scala compiler, which require a lot of efforts for our community to catch up with. Supporting mainstream data lake projects, such as Iceberg, Hudi, and Deltalake is one of the major goals for authz. These projects all heavily depend on private modules of Spark and handle cross-version compatibilities in different ways, or maybe they don't particularly care. Thus, considering the maintenance burden of the Kyuubi community and easy cross-support for data lake projects, it looks like it's time to remove Spark 3.0 in v1.9 for the Spark AuthZ Plugin. Note that for Spark 3.0 users, AuthZ will still bring bug fixes to v1.7.x and v1.8.0. FYI, https://github.com/apache/kyuubi/pull/5363, Yi Zhu is helping with this and comments in this thread and PR 5363 are welcome. Kent
