[
https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264444#comment-14264444
]
Tathagata Das commented on SPARK-4960:
--------------------------------------
Both [[email protected]] and [~jerryshao] are valid. And I looked at the
document, its quite a good proposal. However there are still some corner cases
that is very confusing. What happens when the user accidentally tries to do
something like this
val input = ssc.socketStream(...)
val intercepted = input.interceptor(...)
Now actually use `input` for further processing. Since the `input` stream gets
deregistered, there will not be any data.
To avoid this kind of situation, here is a more limited idea. For the generic
interceptor pattern applicable to all receiver, lets assume that function can
be of the form T => Iterator[T]. This eliminates the need for changing data
types, and probably addresses corner cases like the one I raised. For M
> Interceptor pattern in receivers
> --------------------------------
>
> Key: SPARK-4960
> URL: https://issues.apache.org/jira/browse/SPARK-4960
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Reporter: Tathagata Das
>
> Sometimes it is good to intercept a message received through a receiver and
> modify / do something with the message before it is stored into Spark. This
> is often referred to as the interceptor pattern. There should be general way
> to specify an interceptor function that gets applied to all receivers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
