Eno,

> So in general, we have the problem of failures during writes to the
metadata topic itself.

The KIP suggests to use marker messaged for this case. The marker is
either written (indicating success) or not. If not, after
failure/rebalance the (new) group leader will collect HW again. As long
as the marker is not written, all other instances will not start
processing -- they block until they read the marker which ensures that
the HW is fixed.

>  feel like we are trying to provide some sort of synchronization
between app instances

Not really. Only at application startup, instances reading source topics
need to wait until the leader did write the stop offsets into the
metadata topic (success determined by a marker record). Everything else
is still completely decoupled. The only add-on if for consumer of
intermediate topics -- they don't pick up the stop offsets at start up
but whenever they are available later on.


Does this make sense?


-Matthias

On 11/30/16 2:40 AM, Eno Thereska wrote:
> Hi Matthias,
> 
> I like the first part of the KIP. However, the second part with the failure 
> modes and metadata topic is quite complex and I'm worried it doesn't solve 
> the problems you mention under failure. For example, the application can fail 
> before writing to the metadata topic. In that case, it is not clear what the 
> second app instance should do (for the handling of intermediate topics case). 
> So in general, we have the problem of failures during writes to the metadata 
> topic itself.
> 
> Also, for the intermediate topics example, I feel like we are trying to 
> provide some sort of synchronisation between app instances with this 
> approach. By default today such synchronisation does not exist. One instances 
> writes to the intermediate topic, and the other reads from it, but only 
> eventually. That is a nice way to decouple instances in my opinion. 
> 
> The user can always run the batch processing multiple times and eventually 
> all instances will produce some output. The user's app can check whether the 
> output size is satisfactory and then not run any further loops. So I feel 
> they can already get a lot with the simpler first part of the KIP.
> 
> Thanks
> Eno
> 
> 
>> On 30 Nov 2016, at 05:45, Matthias J. Sax <matth...@confluent.io> wrote:
>>
>> Thanks for your input.
>>
>> To clarify: the main reason to add the metadata topic is to cope with
>> subtopologies that are connected via intermediate topic (either
>> user-defined via through() or internally created for data repartitioning).
>>
>> Without this handling, the behavior would be odd and user experience
>> would be bad.
>>
>> Thus, using the metadata topic for have a "fixed HW" is just a small
>> add-on -- and more or less for free, because the metadata topic is
>> already there.
>>
>>
>> -Matthias
>>
>>
>> On 11/29/16 7:53 PM, Neha Narkhede wrote:
>>> Thanks for initiating this. I think this is a good first step towards
>>> unifying batch and stream processing in Kafka.
>>>
>>> I understood this capability to be simple yet very useful; it allows a
>>> Streams program to process a log, in batch, in arbitrary windows defined by
>>> the difference between the HW and the current offset. Basically, it
>>> provides a simple means for a Streams program to "stop" after processing a
>>> batch, stop (just like a batch program would) and continue where it left
>>> off when restarted. In other words, it allows batch processing behavior for
>>> a Streams app without code changes.
>>>
>>> This feature is useful but I do not think there is a necessity to add a
>>> metadata topic. After all, the user doesn't really care as much about
>>> exactly where the batch ends. This feature allows an app to "process as
>>> much as there is data to process" and the way it determines how much data
>>> there is to process is by reading the HW on startup. If there is new data
>>> written to the log right after it starts up, it will process it when
>>> restarted the next time. If it starts, reads HW but fails, it will restart
>>> and process a little more before it stops again. The fact that the HW
>>> changes in some scenarios isn't an issue since a batch program that behaves
>>> this way doesn't really care exactly what that HW is.
>>>
>>> There might be cases which require adding more topics but I would shy away
>>> from adding complexity wherever possible as it complicates operations and
>>> reduces simplicity.
>>>
>>> Other than this issue, I'm +1 on adding this feature. I think it is pretty
>>> powerful.
>>>
>>>
>>> On Mon, Nov 28, 2016 at 10:48 AM Matthias J. Sax <matth...@confluent.io>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I want to start a discussion about KIP-95:
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> --
>>> Thanks,
>>> Neha
>>>
>>
> 

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to