Hi everyone, just looping in Austin as he mentioned that they also ran into issues due to the inflexibility of the RabiitMQSourcce to me yesterday.
Cheers, Konstantin On Thu, Apr 30, 2020 at 11:23 AM seneg...@gmail.com <seneg...@gmail.com> wrote: > Hello Guys, > > Thanks for all the responses, i want to stress out that i didn't feel > ignored i just thought that i forgot an important step or something. > > Since i am a newbie i would follow whatever route you guys would suggest :) > and i agree that the RMQ connector needs a lot of love still "which i would > be happy to submit gradually" > > as for the code i have it here in the PR: > https://github.com/senegalo/flink/pull/1 it's not that much of a change in > terms of logic but more of what is exposed. > > Let me know how you want me to proceed. > > Thanks again, > Karim Mansour > > On Thu, Apr 30, 2020 at 10:40 AM Aljoscha Krettek <aljos...@apache.org> > wrote: > > > Hi, > > > > I think it's good to contribute the changes to Flink directly since we > > already have the RMQ connector in the respository. > > > > I would propose something similar to the Kafka connector, which takes > > both the generic DeserializationSchema and a KafkaDeserializationSchema > > that is specific to Kafka and allows access to the ConsumerRecord and > > therefore all the Kafka features. What do you think about that? > > > > Best, > > Aljoscha > > > > On 30.04.20 10:26, Robert Metzger wrote: > > > Hey Karim, > > > > > > I'm sorry that you had such a bad experience contributing to Flink, > even > > > though you are nicely following the rules. > > > > > > You mentioned that you've implemented the proposed change already. > Could > > > you share a link to a branch here so that we can take a look? I can > > assess > > > the API changes easier if I see them :) > > > > > > Thanks a lot! > > > > > > > > > Best, > > > Robert > > > > > > On Thu, Apr 30, 2020 at 8:09 AM Dawid Wysakowicz < > dwysakow...@apache.org > > > > > > wrote: > > > > > >> Hi Karim, > > >> > > >> Sorry you did not have the best first time experience. You certainly > did > > >> everything right which I definitely appreciate. > > >> > > >> The problem in that particular case, as I see it, is that RabbitMQ is > > >> not very actively maintained and therefore it is not easy too find a > > >> committer willing to take on this topic. The point of connectors not > > >> being properly maintained was raised a few times in the past on the > ML. > > >> One of the ideas how to improve the situation there was to start a > > >> https://flink-packages.org/ page. The idea is to ask active users of > > >> certain connectors to maintain those connectors outside of the core > > >> project, while giving them a platform within the community where they > > >> can make their modules visible. That way it is possible to overcome > the > > >> lack of capabilities within the core committers without loosing much > on > > >> the visibility. > > >> > > >> I would kindly ask you to consider that path, if you are interested. > You > > >> can of course also wait/reach out to more committers if you feel > strong > > >> about contributing those changes back to the Flink repository itself. > > >> > > >> Best, > > >> > > >> Dawid > > >> > > >> On 30/04/2020 07:29, seneg...@gmail.com wrote: > > >>> Hello, > > >>> > > >>> I am new to the mailing list and to contributing in Big opensource > > >> projects > > >>> in general and i don't know if i did something wrong or should be > more > > >>> patient :) > > >>> > > >>> I put a topic for discussion as per the contribution guide " > > >>> https://flink.apache.org/contributing/how-to-contribute.html"; > almost a > > >> week > > >>> ago and since what i propose is not backward compatible it needs to > be > > >>> discussed here before opening a ticket and moving forward. > > >>> > > >>> So my question is. Will someone pick the discussion up ? or at least > > >>> someone would say that this is not the way to go ? or should i assume > > >> from > > >>> the silence that it's not important / relevant to the project ? > Should > > i > > >>> track the author of the connector and send him directly ? > > >>> > > >>> Thank you for your time. > > >>> > > >>> Regards, > > >>> Karim Mansour > > >>> > > >>> On Fri, Apr 24, 2020 at 11:17 AM seneg...@gmail.com < > > seneg...@gmail.com> > > >>> wrote: > > >>> > > >>>> Dear All, > > >>>> > > >>>> I want to propose a change to the current RabbitMQ connector. > > >>>> > > >>>> Currently the RMQSource is extracting the body of the message which > > is a > > >>>> byte array and pass it to a an instance of a user implementation of > > the > > >>>> DeserializationSchema class to deserialize the body of the message. > It > > >>>> also uses the correlation id from the message properties to > > deduplicate > > >> the > > >>>> message. > > >>>> > > >>>> What i want to propose is instead of taking a implementation of a > > >>>> DeserializationSchema in the RMQSource constructor, actually have > the > > >>>> user implement an interface that would have methods both the output > > for > > >> the > > >>>> RMQSource and the correlation id used not only from the body of the > > >> message > > >>>> but also to it's metadata and properties thus giving the connector > > much > > >>>> more power and flexibility. > > >>>> > > >>>> This of course would mean a breaking API change for the RMQSource > > since > > >> it > > >>>> will no longer take a DeserializationSchema but an implementation > of a > > >>>> predefined interface that has the methods to extract both the output > > of > > >> the > > >>>> RMQSource and the to extract the unique message id as well. > > >>>> > > >>>> The reason behind that is that in my company we were relaying on > > another > > >>>> property the message id for deduplication of the messages and i also > > >> needed > > >>>> that information further down the pipeline and there was absolutely > no > > >> way > > >>>> of getting it other than modifying the RMQSource. > > >>>> > > >>>> I already have code written but as the rules dictates i have to run > it > > >> by > > >>>> you guys first before i attempt to create a Jira ticket :) > > >>>> > > >>>> Let me know what you think. > > >>>> > > >>>> Regards, > > >>>> Karim Mansour > > >>>> > > >> > > >> > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
