Hi Miguel, Of course you can use "Processor API" to achieve what you want. But it needs more coding.
Alternatively, I think you can define a better value for "*max.task.idle.ms <http://max.task.idle.ms>*" configuration. Default value is 0, which means it basically doesn't wait for more data in empty partitions. You can check the doc here <https://kafka.apache.org/30/documentation/streams/developer-guide/config-streams#max-task-idle-ms>, and also the KIP-695 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-695%3A+Further+Improve+Kafka+Streams+Timestamp+Synchronization>, KIP-353 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Synchronization> for more information. Thank you. Luke On Sat, Dec 4, 2021 at 2:10 AM Miguel González <miguel.gonza...@klar.mx> wrote: > Hello > > So I've been using a Streams app to join two input topics... the messages > have a certain order... but I have seen the messages on the output topic > arriving with a different ordering .... Even before, when doing a > map/flatmap operation are processed with different ordering. > > Example: > > Stream 1: A---B---C---D > Stream 2: A--B--C--D > > Output topic: BB---A---AA---CC---DD > > I need it to be A--AA--BB---CC---DD > > Is there a way that Kafka Streams guarantees the order of messages in the > output topic? I had heard when doing a join, the timestamp of the generated > message is taken from the message triggering the join or the biggest > timestamp of the two. I don't know if this is the case, if it's done... is > that an automatic process or do I need to set the timestamp somewhere? > > Another option I have read about is doing something like this: > https://dzone.com/articles/how-to-order-streamed-dataframes where the > author is doing some kind of sorting using an AbstractProcessor, which > seems like a pretty old article. But right I guess I could use a > Transformer or a Processor > > > Any guidance is really appreciated! > > many thanks > - Miguel >
