Elizaveta Lomteva created BEAM-14101:
----------------------------------------
Summary: Design and implement Proxy for Spark Receivers
Key: BEAM-14101
URL: https://issues.apache.org/jira/browse/BEAM-14101
Project: Beam
Issue Type: Task
Components: io-java-cdap
Reporter: Elizaveta Lomteva
h3. Context:
CDAP plugins that support streaming sources include Receiver classes (ex.
[HubSpotReceiver|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotReceiver.java])
that extend {{org.apache.spark.streaming.receiver.Receiver}} abstract class.
Receiver classes are used by plugin Streaming Utils classes (ex.
[HubSpotStreamingUtils|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotStreamingUtil.java])
to provide {{getStream()}} method to Streaming Source classes (ex.
[HubSpotStreamingSource|https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming/HubspotStreamingSource.java])
and usually placed in {{cdap/plugin/\{<plugin name>}/plugin/source/streaming}}
folder (ex.[ HubSpot plugin
repo|https://github.com/data-integrations/hubspot/tree/develop/src/main/java/io/cdap/plugin/hubspot/source/streaming]).
Reference information:
* [Streaming plug-ins
integration|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.o88i6p9b13o9]
(Apache CDAP Connection Design Doc)
* [Plugin integration process
description|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.1h6udb1b52xc]
(Apache CDAP Connection Design Doc)
* [Streaming wrapper
design|https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit#heading=h.fcafz0ydsso1]
(Apache CDAP Connection Design Doc)
h3. Task Description:
Required to design custom Spark receivers proxy class so that the custom
receivers can be used in Apache Beam connector via SparkReceiverIO interface
(used in CDAP IO as a dependency).
h3. Acceptance criteria:
Design of proxy class(es) that will provide custom Spark receivers
functionality to Apache Beam connectors ({{{}SparkReceiverIO{}}}).
h4. Note:
It is necessary that this proxy class be independent of CDAP receivers and can
be used by any other custom Spark receiver.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
