Hi folks, I am working with the new PulsarIO connector with Beam, and most of my work has been in researching how Pulsar works, as many of you know we already have KafkaIO connector which is kind of similar to Pulsar but there is some difference that I have found during my research and I would like to know your input in how would you handle the implementation for SDF. Here are my main concerns: - As you may know kafka handles by default partitioned topics where each message within the partition gets an incremental id, called offset. Having this in mind SDF implementation for kafka works something like this, where the element to evaluate is the topic/partition and the restrictions are the start and end offsets. - For Pulsar, partitioned topics are optional <https://pulsar.apache.org/docs/en/concepts-messaging/#partitioned-topics> or well by default are handled by single broker, there is a possibility where you can use the partitioned topics, but you will limit the final user to use only partitioned topics with pulsar, as well, there is a possibility to manually handle cursors <https://pulsar.apache.org/docs/en/2.5.1/concepts-clients/#reader-interface> which will be the earliest and latest message available that may be used as restrictions (but implementing this will not allow to use partitioned topics). So with this in mind I was thinking there should be two implementations one that use partitioned topics with pulsar and the other one that manually handle cursors.
So, let me know your ideas/input about it. And maybe If i am wrong help to clarify the SDF restrictions for KafkaIO. Thanks, -- *Marco Robles* *|* WIZELINE Software Engineer [email protected] Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
