[
https://issues.apache.org/jira/browse/BAHIR-300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nitin Chauhan updated BAHIR-300:
--------------------------------
Description:
In spark streaming pubsub connector, currently there is no way of getting
source topic from the received messages.
We have a use case where in we are creating multiple streams and taking union
of them in order to process stream from several pubsub topics.
Here we were getting issue to determine which message is coming from which
topic which is required to solve our use case.
Here we have two case
# Topic info was provided during stream creation:
## We can append source topic info in pubsub message attributes before store
it to store.
# Topic info was not provided during stream creation:
## We don't have any source topic info so nothing will be appended.
This Jira is for adding support for adding source as part of PubSubMessage
attributes which is available to be consume during processing RDDs.
was:
In spark streaming pubsub connector, receiver is using reliable store (i.e
store(iterator)) which does not support rate limit and backpressure.
This Jira is for adding rate limit and backpressure in streaming pubsub
connector.
> Source topic info in received message attribute for multi topic stream
> ----------------------------------------------------------------------
>
> Key: BAHIR-300
> URL: https://issues.apache.org/jira/browse/BAHIR-300
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
> Reporter: Nitin Chauhan
> Priority: Minor
> Labels: GCP, pubsub, spark
>
> In spark streaming pubsub connector, currently there is no way of getting
> source topic from the received messages.
> We have a use case where in we are creating multiple streams and taking union
> of them in order to process stream from several pubsub topics.
> Here we were getting issue to determine which message is coming from which
> topic which is required to solve our use case.
>
> Here we have two case
> # Topic info was provided during stream creation:
> ## We can append source topic info in pubsub message attributes before store
> it to store.
> # Topic info was not provided during stream creation:
> ## We don't have any source topic info so nothing will be appended.
>
> This Jira is for adding support for adding source as part of PubSubMessage
> attributes which is available to be consume during processing RDDs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
