Hi Qing again, Another point to consider: broadcast streams are subject to watermarking. i.e.
* You can wait to process the broadcast records only after the watermark passed, then * order those records by time * keep all broadcast records where the watermark not yet passed in some extra data structure without processing them * that also means, the broadcast stream should not be configured to use watermark idleness, or manually implement watermark processing logic for the idling broadcast stream This sounds a little complicated, but can definitely be done (I do that all the time 😊 ) Best regards Thias From: 仙路尽头谁为峰 <xljtswf2...@163.com> Sent: Wednesday, October 5, 2022 10:13 AM To: Qing Lim <q....@mwam.com> Cc: User <user@flink.apache.org> Subject: [SPAM] 回复: Re:Question about Flink Broadcast State event ordering ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hi Qing: The key point is that the broadcast side may have different partitions that interleaves. If you can make sure those messages you want to be ordered go into the same partition, then I think the order can be reserved. Best regards! 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 发件人: Qing Lim<mailto:q....@mwam.com> 发送时间: 2022年10月5日 15:16 收件人: xljtswf2022<mailto:xljtswf2...@163.com> 抄送: User<mailto:user@flink.apache.org> 主题: RE: Re:Question about Flink Broadcast State event ordering Hi, thanks for answering my question. Is there anyway to make the order reflecting the upstream? I wish to broadcast messages that has deletion semantic, so ordering matters here. I guess worst case I can use some logical timestamp to reason about order at downstream. From: xljtswf2022 <xljtswf2...@163.com<mailto:xljtswf2...@163.com>> Sent: 05 October 2022 03:02 To: Qing Lim <q....@mwam.com<mailto:q....@mwam.com>> Cc: User <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re:Question about Flink Broadcast State event ordering Hi Qing: > I think this is refering to the order between broadcasted element and non > broadcasted element, right? No, as broadcast and nonbroadcast stream are different streams, they will usually transfer with different tcp connection, we can not control the order of elements in different connections. > The broadcasted element should arrive in the same order across all tasks, > right? no. image the broadcast stream has 2 partitions ,say p1 and p2. and each partition has elements with index 1, 2, 3.... then one downstream task may see the broadcast stream p1-1, p1-2 ..... p2-1, p2-2... and another will see p1-1, p2-1,p1-2,p2-2.... ps: for elements usually come in bulk, the index is just for explanation. Best regards! At 2022-10-04 21:54:23, "Qing Lim" <q....@mwam.com<mailto:q....@mwam.com>> wrote: Hi Flink user group, I have a question around broadcast. Reading the docs https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#important-considerations, it says the following: > Order of events in Broadcast State may differ across tasks: Although > broadcasting the elements of a stream guarantees that all elements will > (eventually) go to all downstream tasks, elements may arrive in a different > order to each task. So the state updates for each incoming element MUST NOT > depend on the ordering of the incoming events. I think this is refering to the order between broadcasted element and non broadcasted element, right? The broadcasted element should arrive in the same order across all tasks, right? For example, given a broadcasted stream A, and a non-broadcasted stream B When joining A and B, elements from A should always reach all tasks in the same order right? Its just the interleaving of A and B that might differ across tasks, did I understand it correctly? I wasn’t sure because its not clear to me by just reading the doc, happy to update the doc once its clarified here. Kind regards. This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession. To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations. To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email. Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training. Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
