IIUC, your scenario is quite like what currently ReliableKafkaReceiver does. You can only send ack to the upstream source after WAL is persistent, otherwise because of asynchronization of data processing and data receiving, there's still a chance data could be lost if you send out ack before WAL.
You could refer to ReliableKafkaReceiver. On Thu, Mar 9, 2017 at 12:58 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I am using a Receiver based approach. And I understand that spark > streaming API's will convert the received data from receiver into blocks > and these blocks that are in memory are also stored in WAL if one enables > it. my upstream source which is not Kafka can also replay by which I mean > if I don't send an ack to my upstream source it will resend it so I don't > have to write the received data to WAL however I still need to enable WAL > correct? because there are blocks that are in memory that needs to written > to WAL so they can be recovered later. > > Thanks, > kant >
