[
https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14252146#comment-14252146
]
Cody Koeninger commented on SPARK-3146:
---------------------------------------
>From my point of view, the interceptor pattern is little more than an overly
>complex way of spelling flatMap. What does it actually gain you?
The specific issue here is two-fold:
1. Exposing MessageAndMetadata to users, not just (key, message). This is a
kafka-specific concern, not applicable to all types of receivers. If it had
been originally written as a DStream[MessageAndMetadata], there would be less
of a need for this patch.
2. Controlling timing, and thus checkpointing, based on the contents of the
individual item. I can see this being applicable to many types of receivers,
but not all. For instance, if I sleep inside this function, what happens for
Kafka is not the same as what happens for e.g. Twitter.
> Improve the flexibility of Spark Streaming Kafka API to offer user the
> ability to process message before storing into BM
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-3146
> URL: https://issues.apache.org/jira/browse/SPARK-3146
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Saisai Shao
>
> Currently Spark Streaming Kafka API stores the key and value of each message
> into BM for processing, potentially this may lose the flexibility for
> different requirements:
> 1. currently topic/partition/offset information for each message is discarded
> by KafkaInputDStream. In some scenarios people may need this information to
> better filter the message, like SPARK-2388 described.
> 2. People may need to add timestamp for each message when feeding into Spark
> Streaming, which can better measure the system latency.
> 3. Checkpointing the partition/offsets or others...
> So here we add a messageHandler in interface to give people the flexibility
> to preprocess the message before storing into BM. In the meantime time this
> improvement keep compatible with current API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
