[
https://issues.apache.org/jira/browse/SPARK-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14197464#comment-14197464
]
Hari Shreedharan commented on SPARK-4174:
-----------------------------------------
I will write up a doc soon, but here are some initial thoughts:
We should be able to provide one callback per batch, unfortunately receivers
are not really batch aware. So there are couple of options:
- Take a function to be called per block, and call all of the functions in
order. This can generate as many callbacks as the blocks per batch - this may
not be huge but this also means that there is a chain of methods that would get
called when each block gets completed. This is most likely the simplest as far
as semantics are concerned.
- We could add a new method to the Receiver API -
`onBlockComplete(List[BlockId])`. This would allow the receiver to implement
Unfortunately none of the push* methods returned the block id - therefore this
method would really be useful only if the receiver calls the push* methods with
the block id passed in. We'd have to document this though.
I am open to other semantics as well, but the API is a bit limiting since we
don't report the block info back to the receiver as of now.
The implementation idea I have for either would bepretty similar. Before I go
into that, any thoughts on the semantics?
> Streaming: Optionally provide notifications to Receivers when DStream has
> been generated
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-4174
> URL: https://issues.apache.org/jira/browse/SPARK-4174
> Project: Spark
> Issue Type: Improvement
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
>
> Receivers receiving data from Message Queues, like Active MQ, Kafka etc can
> replay messages if required. Using the HDFS WAL mechanism for such systems
> affects efficiency as we are incurring an unnecessary HDFS write when we can
> recover the data from the queue anyway.
> We can fix this by providing a notification to the receiver when the RDD is
> generated from the blocks. We need to consider the case where a receiver
> might fail before the RDD is generated and come back on a different executor
> when the RDD is generated. Either way, this is likely to cause duplicates and
> not data loss -- so we may be ok.
> I am thinking about something of the order of accepting a callback function
> which gets called when the RDD is generated. We can keep the function local
> in a map of batch id -> function, which gets called when the function gets
> generated (we can inform the ReceiverSupervisorImpl via Akka when the driver
> generates the RDD). Of course, just an early thought - I will work on a
> design doc for this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
