Wellington Chevreuil created HBASE-30178:
--------------------------------------------
Summary: Add support to spark 4 in hbase-connectors
Key: HBASE-30178
URL: https://issues.apache.org/jira/browse/HBASE-30178
Project: HBase
Issue Type: Umbrella
Components: hbase-connectors
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil
With spark 4.0.0 GA for around a year now, we should start working on making
hbase-connectors integrate with spark 4.
Since we would still need to support spark 3 for a while and due to spark 4
requiring minimum jdk17 and scala 2.13, it would be easier to define a specific
spark 4 module under the hbase-connectors root, so that spark.version,
scala.version, compiler flags, and optional spark-4 only APIs never collide
with spark 3 in a single parent pom.
We should keep feature parity as much as possible. Datasources, context, rdd
and connection cache related logic can be duplicated under this new
hbase-spark4 module, in order to compile and package with jdk17 and scala 2.13.
This module build should be isolated on a specific maven profile, so it
wouldn't be build by default.
Pushdown filter related classes and filter protocol would remain on a common
module (no code duplication needed).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
