[
https://issues.apache.org/jira/browse/HBASE-30399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil resolved HBASE-30399.
------------------------------------------
Resolution: Fixed
> Add spark4 compatible streaming functionality
> ---------------------------------------------
>
> Key: HBASE-30399
> URL: https://issues.apache.org/jira/browse/HBASE-30399
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> The spark3 module has streaming via the deprecated DStream/StreamingContext
> API (HBaseDStreamFunctions: hbaseBulkPut, hbaseBulkDelete, hbaseBulkGet,
> hbaseForeachPartition, hbaseMapPartitions).
> This is to port the spark3 streaming capabilities using Spark 4's Structured
> Streaming (DS V2) only. Additionally to port the sink implementation, we
> should leverage Spark4 Structured Streaming MicroBatch and ContinuousStream
> sources.
> The plan is to implement these as separate PRs, for further granularity and
> ease of reviews:
> 1) Streaming sink — replaces spark3 hbaseBulkPut/hbaseBulkDelete
> (STREAMING_WRITE)
> -2) MicroBatch source — new capability (no spark3 equivalent) using HBase
> cell timestamps as offsets (MICRO_BATCH_READ)-
> -3) ContinuousStream source — new capability, implemented as a polling loop
> over HBase (CONTINUOUS_READ)-
> The enrichment-lookup pattern from spark3 (hbaseBulkGet/hbaseMapPartitions)
> is covered without new classes: users call foreachBatch and use the existing
> batch read API to join/enrich. Ex:
> {code:java}
> val stream = spark.readStream.format("kafka")...load()
> val hbaseLookup = spark.read.format("hbase").option("catalog", ...).load()
> val enriched = stream.join(hbaseLookup, stream("key") === hbaseLookup("id"))
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
