[
https://issues.apache.org/jira/browse/HBASE-30399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-30399:
-----------------------------------------
Description:
The spark3 module has streaming via the deprecated DStream/StreamingContext API
(HBaseDStreamFunctions: hbaseBulkPut, hbaseBulkDelete, hbaseBulkGet,
hbaseForeachPartition, hbaseMapPartitions).
This is to port the spark3 streaming capabilities using Spark 4's Structured
Streaming (DS V2) only. Additionally to port the sink implementation, we should
leverage Spark4 Structured Streaming MicroBatch and ContinuousStream sources.
The plan is to implement these as separate PRs, for further granularity and
ease of reviews:
1) Streaming sink — replaces spark3 hbaseBulkPut/hbaseBulkDelete
(STREAMING_WRITE)
2) MicroBatch source — new capability (no spark3 equivalent) using HBase cell
timestamps as offsets (MICRO_BATCH_READ)
3) ContinuousStream source — new capability, implemented as a polling loop over
HBase (CONTINUOUS_READ)
The enrichment-lookup pattern from spark3 (hbaseBulkGet/hbaseMapPartitions) is
covered without new classes: users call foreachBatch and use the existing batch
read API to join/enrich.
> Add spark4 compatible streaming functionality
> ---------------------------------------------
>
> Key: HBASE-30399
> URL: https://issues.apache.org/jira/browse/HBASE-30399
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> The spark3 module has streaming via the deprecated DStream/StreamingContext
> API (HBaseDStreamFunctions: hbaseBulkPut, hbaseBulkDelete, hbaseBulkGet,
> hbaseForeachPartition, hbaseMapPartitions).
> This is to port the spark3 streaming capabilities using Spark 4's Structured
> Streaming (DS V2) only. Additionally to port the sink implementation, we
> should leverage Spark4 Structured Streaming MicroBatch and ContinuousStream
> sources.
> The plan is to implement these as separate PRs, for further granularity and
> ease of reviews:
> 1) Streaming sink — replaces spark3 hbaseBulkPut/hbaseBulkDelete
> (STREAMING_WRITE)
> 2) MicroBatch source — new capability (no spark3 equivalent) using HBase cell
> timestamps as offsets (MICRO_BATCH_READ)
> 3) ContinuousStream source — new capability, implemented as a polling loop
> over HBase (CONTINUOUS_READ)
> The enrichment-lookup pattern from spark3 (hbaseBulkGet/hbaseMapPartitions)
> is covered without new classes: users call foreachBatch and use the existing
> batch read API to join/enrich.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
