[
https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265508#comment-14265508
]
Saisai Shao commented on SPARK-4960:
------------------------------------
Hi TD, thanks a lot for your suggestions. What I concerned about changing the
interceptor pattern to T => Iterator\[T\] is that it will narrow the usage
scenario of this functionality, at least it cannot meet the requirement
[[email protected]] mentioned for KafkaInputDStream. Is it necessary to add
just T => Iterator\[T\] pattern in receiver rather than moving into DStream
transformation?
For the problem you mentioned, I think maybe we can create a interceptor
receiver to ship the data from actual receiver and do some transformation
before storing, so each actual receiver will have a shadow interceptor receiver
to intercept the data, this might fix the problem but will increase the
implementation complexity.
I will re-think about my design and try to figure out a better way if possible.
Thanks a lot.
> Interceptor pattern in receivers
> --------------------------------
>
> Key: SPARK-4960
> URL: https://issues.apache.org/jira/browse/SPARK-4960
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Reporter: Tathagata Das
>
> Sometimes it is good to intercept a message received through a receiver and
> modify / do something with the message before it is stored into Spark. This
> is often referred to as the interceptor pattern. There should be general way
> to specify an interceptor function that gets applied to all receivers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
