[
https://issues.apache.org/jira/browse/HBASE-30189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-30189:
-----------------------------------------
Description:
The spark4 module currently has basic HBase RDD operations (put, get, delete,
foreachPartition, mapPartitions). The next step is porting the SparkSQL
datasource layer from the spark3 module, which enables reading/writing HBase
tables via Spark SQL (spark.read.format("hbase")...). The spark3 module uses
the deprecated DataSource V1 API (RelationProvider, BaseRelation,
PrunedFilteredScan).
For spark4, we'll implement the modern DataSource V2 API, which provides better
performance (columnar reads, push-down at the connector level) and a cleaner
contract.
In order to have granular, incremental PRs for easy of reviews, I'll split this
in smaller PRs to be merged into a feature branch first. Once this is complete,
we can cherry-pick the commits into master.
was:
The spark4 module currently has basic HBase RDD operations (put, get, delete,
foreachPartition, mapPartitions). The next step is porting the SparkSQL
datasource layer from the spark3 module, which enables reading/writing HBase
tables via Spark SQL (spark.read.format("hbase")...). The spark3 module uses
the deprecated DataSource V1 API (RelationProvider, BaseRelation,
PrunedFilteredScan).
For spark4, we'll implement the modern DataSource V2 API, which provides better
performance (columnar reads, push-down at the connector level) and a cleaner
contract.
> Implement SparkSQL support in the spark4 module using DataSource V2 API
> -----------------------------------------------------------------------
>
> Key: HBASE-30189
> URL: https://issues.apache.org/jira/browse/HBASE-30189
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> The spark4 module currently has basic HBase RDD operations (put, get, delete,
> foreachPartition, mapPartitions). The next step is porting the SparkSQL
> datasource layer from the spark3 module, which enables reading/writing HBase
> tables via Spark SQL (spark.read.format("hbase")...). The spark3 module uses
> the deprecated DataSource V1 API (RelationProvider, BaseRelation,
> PrunedFilteredScan).
> For spark4, we'll implement the modern DataSource V2 API, which provides
> better performance (columnar reads, push-down at the connector level) and a
> cleaner contract.
> In order to have granular, incremental PRs for easy of reviews, I'll split
> this in smaller PRs to be merged into a feature branch first. Once this is
> complete, we can cherry-pick the commits into master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
