Siyuan Hua commented on APEXMALHAR-2283:

There are couple of solutions to "exactly-once". To me they are all different 
and they all have different assumptions. 
First of all, if we assume there is unique message id, we can definitely use 
that for dedup, but that is not always the case, then we need appid and 
operatorid to do dedup.
Then this information can be store in either key or extra topic, I wouldn't say 
either of them is better than the other, really depends on how user's 
And no matter what we do, as long as there are number of operators writes to 
same kafka partition, I'm afraid there is no way to do perfect dedup because we 
don't know the safe place to do dedup from or too late to do dedup(too much 
noise from the safe place if other operator instances are fully loaded)
I don't remember hashcode, but I think hashcode will include some false 

And there is other solution like, create topic and number of partitions 
automatically based on kafka operator instances, in this case, it is much 
easier, we are always use what messages needs to be dedup because one operator 
only write to one kafka partition. This solution, my understanding, is most 
reliable and some user might want it. But the metadata of that kafka topic is 
kind of automatic created and it's very hard to support dynamic partition in 
this case.

That's the reason why I say there is no general solution for exactly-once kafka 
output operator. We may need to provide different solutions in "examples" for 
people to choose from. 

Anyways, Sandesh, can you wrap up the current solution, post and discuss it in 
mailing list?


> Refactor kafka output operator
> ------------------------------
>                 Key: APEXMALHAR-2283
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2283
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: Siyuan Hua
>            Assignee: Siyuan Hua
> The abstract kafka output operator needs to be refactored
> 1. Needs to set some mandatory properties on operator level instead of kafka 
> property level.
> 2. More document and examples
> 3. Find a standard way to achieve exactly once in both 0.8 and 0.9
> More will be added when working on the ticket

This message was sent by Atlassian JIRA

Reply via email to