Hi all, I made my first pull request [1]. Since this project is brand new (no parent poms, licensing), I reckoned it was best to use Saikat's branch on Flink. I suppose the Flink branch will be merged first. If not please give me guidelines on how I should proceed next.
Kind regards Emmanouil [1] https://github.com/apache/ignite-extensions/pull/2 *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Fri, 22 Nov 2019 at 20:55, Denis Magda <dma...@apache.org> wrote: > Awesome, ping us whenever you're ready! > > - > Denis > > > On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > wrote: > > > Hi all! > > > > I am sorry for being late on that. I was trying to refactor the test in > > order not to be in need of any external tools or spinning up a server. > > I did forked the new repo and indeed my changes there, so a pull request > is > > a matter of time! > > > > Kind regards > > > > *Emmanouil Gkatziouras* > > https://egkatzioura.com/ | > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > https://github.com/gkatzioura > > > > > > On Fri, 22 Nov 2019 at 20:45, Denis Magda <dma...@apache.org> wrote: > > > > > Hi Emmanouil, > > > > > > Do you have any questions or need any support from the community? > > > > > > - > > > Denis > > > > > > > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra <saikat.mai...@gmail.com > > > > > wrote: > > > > > >> Hi Emmanouil, > > >> > > >> Can you please take a looks at dev utils, if this is something you are > > >> looking for > > >> https://github.com/apache/ignite/tree/master/modules/dev-utils? > > >> > > >> IMO, if you can release Pub/Sub server in maven and then use it as > > >> dependency, that would be great. > > >> > > >> Regards, > > >> Saikat > > >> > > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra < > saikat.mai...@gmail.com> > > >> wrote: > > >> > > >> > Hi Emmanouil, > > >> > > > >> > The master branch has been initialized. I have raised a PR to > migrate > > >> > Flink module to ignite-extensions. > > >> > > > >> > PR https://github.com/apache/ignite-extensions/pull/1 > > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > > >> > > > >> > Regards, > > >> > Saikat > > >> > > > >> > > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < > > >> > gkatzio...@gmail.com> wrote: > > >> > > > >> >> Hi all! > > >> >> > > >> >> This is an update. The integration is tested with mocks based on > the > > >> >> original Ignite repo [1]. > > >> >> I am currently waiting for the first commits on the new integration > > >> >> project > > >> >> to take place in order to create my pull request accordingly to the > > >> >> initial > > >> >> commits. > > >> >> In order to simulate the Pub/Sub server in a test I used a test > > server > > >> >> implementation [2]. Unfortunately they don't have this util > uploaded > > >> in a > > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache > > License > > >> >> 2.0). Is there a repo for utilities that Ignite uses for the > > >> test/build? > > >> >> I have also created a ticket [3] > > >> >> > > >> >> Kind regards > > >> >> > > >> >> [1] > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > >> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator > > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 > > >> >> > > >> >> *Emmanouil Gkatziouras* > > >> >> https://egkatzioura.com/ | > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> https://github.com/gkatzioura > > >> >> > > >> >> > > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra < > saikat.mai...@gmail.com > > > > > >> >> wrote: > > >> >> > > >> >> > Hello Ilya, > > >> >> > > > >> >> > Thank you for your email. Yes, before we remove any integration > we > > >> will > > >> >> > move them to another repository and release. > > >> >> > > > >> >> > We are thinking we will start with Pub/Sub Streamer > implementation > > >> and > > >> >> > continue to move other integration as mentioned in the following > > doc. > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >> >> > > > >> >> > Regards, > > >> >> > Saikat > > >> >> > > > >> >> > > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < > > >> >> > ilya.kasnach...@gmail.com> > > >> >> > wrote: > > >> >> > > > >> >> > > Hello! > > >> >> > > > > >> >> > > My take on this: we want to offload some of integrations, but > > >> please > > >> >> > commit > > >> >> > > and release them to another project first, and only then we > will > > >> >> decide > > >> >> > to > > >> >> > > remove them from our repository. > > >> >> > > > > >> >> > > Please also make sure they are available with same artifact > name > > >> and > > >> >> API. > > >> >> > > If this has to be changed, let's wait for AI 3.0 > > >> >> > > > > >> >> > > Regards, > > >> >> > > -- > > >> >> > > Ilya Kasnacheev > > >> >> > > > > >> >> > > > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra < > > >> saikat.mai...@gmail.com>: > > >> >> > > > > >> >> > > > Hello Emmanouil, > > >> >> > > > > > >> >> > > > As discussed earlier, I discussed with Apache Bahir community > > and > > >> >> they > > >> >> > > are > > >> >> > > > interested to have Apache Ignite extensions as part of Apache > > >> Bahir > > >> >> > > > project. > > >> >> > > > > > >> >> > > > Can you please share Pub/Sub streamer implementation details > > with > > >> >> > Apache > > >> >> > > > Bahir community and request for feedback. > > >> >> > > > > > >> >> > > > https://bahir.apache.org/ > > >> >> > > > > > >> >> > > > I have also requested for contributor access in Jira for > Apache > > >> >> Bahir > > >> >> > > > project so that I can create issues and assign to myself. > > >> >> > > > > > >> >> > > > I can help with code reviews as well. > > >> >> > > > > > >> >> > > > Here is the email thread for reference > > >> >> > > > > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > >> >> > > > > > >> >> > > > Regards, > > >> >> > > > Saikat > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra < > > >> >> saikat.mai...@gmail.com > > >> >> > > > > >> >> > > > wrote: > > >> >> > > > > > >> >> > > > > Hello, > > >> >> > > > > > > >> >> > > > > As discussed I have created following discussion threads on > > our > > >> >> > > migration > > >> >> > > > > proposals for Apache Ignite extensions: > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html > > >> >> > > > > > > >> >> > > > > > > >> https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html > > >> >> > > > > > > >> >> > > > > I will share update as I hear more information. > > >> >> > > > > > > >> >> > > > > Regards, > > >> >> > > > > Saikat > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra < > > >> >> > saikat.mai...@gmail.com > > >> >> > > > > > >> >> > > > > wrote: > > >> >> > > > > > > >> >> > > > >> Hello Denis, > > >> >> > > > >> > > >> >> > > > >> Sure, sounds good. I will create a separate discussion > > thread > > >> to > > >> >> get > > >> >> > > > >> community feedback on whether we > > >> >> > > > >> should join the Bahir or kick-off "Ignite Extensions" > > project. > > >> >> > > > >> > > >> >> > > > >> Regards, > > >> >> > > > >> Saikat > > >> >> > > > >> > > >> >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda < > > >> dma...@apache.org> > > >> >> > > wrote: > > >> >> > > > >> > > >> >> > > > >>> Folks, > > >> >> > > > >>> > > >> >> > > > >>> The concept of Apache Bahir is precisely what we need! > > >> Saikat, > > >> >> > thanks > > >> >> > > > for > > >> >> > > > >>> doing the research. Why I believe the * idea* fits us > well: > > >> >> > > > >>> > > >> >> > > > >>> - All integrations can be stored in separate Github > > >> >> repositories > > >> >> > > and > > >> >> > > > >>> have their dev lifecycles. We've not obliged to couple > > all > > >> >> the > > >> >> > > > >>> integrations > > >> >> > > > >>> in a single repo. For instance, Spark can be located > in > > a > > >> >> > > dedicated > > >> >> > > > >>> repo > > >> >> > > > >>> while streaming integrations might be in a single one. > > >> It's > > >> >> up > > >> >> > to > > >> >> > > > us. > > >> >> > > > >>> - All the repositories and integrations belong to ASF. > > >> We're > > >> >> not > > >> >> > > > >>> dumping > > >> >> > > > >>> them on Github but rather keep supporting and > developing > > >> in > > >> >> > > > >>> accordance with > > >> >> > > > >>> ASF vision and practices. > > >> >> > > > >>> - There is a way to reward contributors via > > committership > > >> and > > >> >> > PMC > > >> >> > > > >>> membership. You won't get this if a project is just > one > > of > > >> >> the > > >> >> > > > >>> millions of > > >> >> > > > >>> Github projects. > > >> >> > > > >>> > > >> >> > > > >>> Saikat, as Ignite PMC member, could you please kick-off a > > >> >> separate > > >> >> > > > >>> dev-list > > >> >> > > > >>> discussion to involve more community members and > referring > > to > > >> >> this > > >> >> > > > >>> thread? > > >> >> > > > >>> I think the primary question the community needs to > answer > > >> >> whether > > >> >> > we > > >> >> > > > >>> should join the Bahir or kick-off "Ignite Extensions" > > >> project? > > >> >> > > > >>> > > >> >> > > > >>> - > > >> >> > > > >>> Denis > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < > > >> >> > > > zaleslaw....@gmail.com> > > >> >> > > > >>> wrote: > > >> >> > > > >>> > > >> >> > > > >>> > Maybe we could move all our Streaming Integrations > there, > > >> but > > >> >> > what > > >> >> > > is > > >> >> > > > >>> about > > >> >> > > > >>> > maintaining and committer permissions to the new > > >> repositories? > > >> >> > > > >>> > I see the list of the committers and PMC members there > > >> >> > > > >>> > https://bahir.apache.org/community-members/ > > >> >> > > > >>> > > > >> >> > > > >>> > Could somebody from Ignite community be added to this > > list > > >> as > > >> >> a > > >> >> > > > >>> > PMC/committer to maintain Ignite integrations? > > >> >> > > > >>> > If yes, I'd happy to join this project and collaborate > > with > > >> >> these > > >> >> > > > guys > > >> >> > > > >>> > together > > >> >> > > > >>> > If no, and we should start from the zero level with > > >> external > > >> >> PRs > > >> >> > - > > >> >> > > > >>> hmmm, > > >> >> > > > >>> > it's better to have our own external repository with > > >> >> ApacheBahirr > > >> >> > > > >>> ideology. > > >> >> > > > >>> > > > >> >> > > > >>> > Also, I agree, that all connectors could be moved there > > (in > > >> >> > > > >>> ApacheBahirr > > >> >> > > > >>> > like repository), but not all modules from the page > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >> >> > > > >>> > because > > >> >> > > > >>> > they use different parts of core modules and could be > > >> >> developed > > >> >> > > with > > >> >> > > > >>> > different velocity. > > >> >> > > > >>> > > > >> >> > > > >>> > In reality, before creation of new repositories we need > > >> wide > > >> >> > > > discussion > > >> >> > > > >>> > based on the document mentioned above. > > >> >> > > > >>> > > > >> >> > > > >>> > P.S. Of course, we could experiment in the next release > > 2.8 > > >> >> with > > >> >> > > one > > >> >> > > > >>> or two > > >> >> > > > >>> > integrations like twitter or ZeroMQ > > >> >> > > > >>> > Also, I'd like Denis Magda idea for the separate > > >> repositories > > >> >> in > > >> >> > > > Apache > > >> >> > > > >>> > Github to maintain them by anybody. > > >> >> > > > >>> > > > >> >> > > > >>> > > > >> >> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra < > > >> >> > > saikat.mai...@gmail.com > > >> >> > > > >: > > >> >> > > > >>> > > > >> >> > > > >>> > > Hi Denis, Emmanouil > > >> >> > > > >>> > > > > >> >> > > > >>> > > Thank you for your email. I think creating a separate > > >> >> > integration > > >> >> > > > >>> project > > >> >> > > > >>> > > in github and also proposing it as an apache > incubator > > >> >> project > > >> >> > > will > > >> >> > > > >>> be a > > >> >> > > > >>> > > good move. The other separate integration modules > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > >> >> > > > >>> > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >> >> > > > >>> > > can > > >> >> > > > >>> > > be moved to this new apache incubator project. > > >> >> > > > >>> > > > > >> >> > > > >>> > > There are similar integration available for Flink and > > >> Spark > > >> >> in > > >> >> > > > Apache > > >> >> > > > >>> > Bahir > > >> >> > > > >>> > > https://bahir.apache.org/ > > >> >> > > > >>> > > > > >> >> > > > >>> > > Do you think we should reach out to Apache Bahir > > >> community > > >> >> > with a > > >> >> > > > >>> > proposal > > >> >> > > > >>> > > or we should plan to create a separate Apache > Incubator > > >> >> > project? > > >> >> > > > >>> > > > > >> >> > > > >>> > > Regards, > > >> >> > > > >>> > > Saikat > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil > Gkatziouras < > > >> >> > > > >>> > > gkatzio...@gmail.com> > > >> >> > > > >>> > > wrote: > > >> >> > > > >>> > > > > >> >> > > > >>> > > > Hi all! > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > Based on the discussions most probably this is > going > > to > > >> >> be on > > >> >> > > > >>> another > > >> >> > > > >>> > > > GitHub repo. Therefore I will prepare a standalone > > >> project > > >> >> > with > > >> >> > > > the > > >> >> > > > >>> > > Pub/Sub > > >> >> > > > >>> > > > module and once the repository is created a pull > > >> request > > >> >> > shall > > >> >> > > be > > >> >> > > > >>> > issued. > > >> >> > > > >>> > > > If there is anything else I can do in order to > > >> facilitate > > >> >> > this > > >> >> > > > >>> please > > >> >> > > > >>> > let > > >> >> > > > >>> > > > me know. > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > If a ticket is created for this issue, I shall be > > able > > >> to > > >> >> > > create > > >> >> > > > a > > >> >> > > > >>> pull > > >> >> > > > >>> > > > request and thus a build will take place on > ignite's > > >> CI. > > >> >> > > > >>> > > > If everyone is aligned with adding a Pub/Sub > > >> >> implementation > > >> >> > > may I > > >> >> > > > >>> > create > > >> >> > > > >>> > > a > > >> >> > > > >>> > > > ticket on JIRA? > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > My next question has to do on how Ignite configures > > the > > >> >> build > > >> >> > > > jobs > > >> >> > > > >>> in > > >> >> > > > >>> > > Team > > >> >> > > > >>> > > > City. The unit tests for my implementation need a > > local > > >> >> > Pub/Sub > > >> >> > > > >>> server > > >> >> > > > >>> > up > > >> >> > > > >>> > > > and running (they provide one for testing > purposes). > > >> Will > > >> >> > this > > >> >> > > > take > > >> >> > > > >>> > > effect > > >> >> > > > >>> > > > in a form of a build script inside my > implementation > > or > > >> >> is it > > >> >> > > > >>> something > > >> >> > > > >>> > > > that should be configured on Team City? > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > Kind regards, > > >> >> > > > >>> > > > Emmanouil > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > *Emmanouil Gkatziouras* > > >> >> > > > >>> > > > https://egkatzioura.com/ | > > >> >> > > > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> > > > >>> > > > https://github.com/gkatzioura > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda < > > >> >> dma...@apache.org > > >> >> > > > > >> >> > > > >>> wrote: > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > Emmanouil, Saikat, > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > After contemplating on this, let me propose > another > > >> way > > >> >> > with > > >> >> > > > this > > >> >> > > > >>> > > > > particular integration that is aligned with our > > >> >> > > modularization > > >> >> > > > >>> > > endeavor. > > >> >> > > > >>> > > > > The modularization [1] defines the Ignite core, > > >> Ignite > > >> >> > > modules, > > >> >> > > > >>> and > > >> >> > > > >>> > > > > independent integrations. The first two are to be > > >> >> located > > >> >> > in > > >> >> > > > >>> Ignite > > >> >> > > > >>> > > > > repositories and to be tested/updated by the > > >> community > > >> >> for > > >> >> > > > every > > >> >> > > > >>> > > release. > > >> >> > > > >>> > > > > The modules are a subset of existing integrations > > >> that > > >> >> are > > >> >> > > > >>> adopted > > >> >> > > > >>> > > widely > > >> >> > > > >>> > > > > and require community attention. > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > As for the independent integrations [1], those > are > > >> >> supposed > > >> >> > > to > > >> >> > > > be > > >> >> > > > >>> > > located > > >> >> > > > >>> > > > > in a separate GitHub repo. I would advise us to > > start > > >> >> > > > >>> implementing > > >> >> > > > >>> > this > > >> >> > > > >>> > > > > concept with the Pub/Sub integration. Even though > > it > > >> >> will > > >> >> > be > > >> >> > > > >>> located > > >> >> > > > >>> > > in a > > >> >> > > > >>> > > > > separate Github repo, it will be listed among > > >> officially > > >> >> > > > >>> available > > >> >> > > > >>> > > > > integrations on the Ignite website featuring > > >> Emmanouil > > >> >> as a > > >> >> > > > >>> > > contributor. > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > What do you think? > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > [1] > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > >> >> > > > >>> > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > - > > >> >> > > > >>> > > > > Denis > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < > > >> >> > > > >>> > saikat.mai...@gmail.com > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > wrote: > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > Hi Emmanouil, > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > The changes looks good to me. I wanted to check > > if > > >> you > > >> >> > are > > >> >> > > > also > > >> >> > > > >>> > able > > >> >> > > > >>> > > to > > >> >> > > > >>> > > > > > validate that the once data is added to Ignite > > >> cluster > > >> >> > you > > >> >> > > > are > > >> >> > > > >>> also > > >> >> > > > >>> > > > able > > >> >> > > > >>> > > > > to > > >> >> > > > >>> > > > > > access it using another Ignite client or using > > rest > > >> >> > > > endpoints. > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > We use Teamcity for CI process, the details are > > as > > >> >> > follows > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > Trigger validation of those test suites that > have > > >> been > > >> >> > > > >>> affected by > > >> >> > > > >>> > > your > > >> >> > > > >>> > > > > > changes on TeamCity < > > http://ci.ignite.apache.org/>: > > >> < > > >> >> > > > >>> > > > > http://204.14.53.153/ > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > - Locate a test suite you have to check, > press > > >> >> button > > >> >> > > > named > > >> >> > > > >>> > "..." > > >> >> > > > >>> > > > that > > >> >> > > > >>> > > > > > is located on the left of "Run" button. "Run > > >> custom > > >> >> > > build" > > >> >> > > > >>> > dialog > > >> >> > > > >>> > > > will > > >> >> > > > >>> > > > > > appear; > > >> >> > > > >>> > > > > > - Go to "Changes" tab and choose > > >> >> > > > >>> > "pull/<pull-request-number>/head" > > >> >> > > > >>> > > > in > > >> >> > > > >>> > > > > > "Build branch" dropdown list; > > >> >> > > > >>> > > > > > - Press "Run build" button and monitor tests > > >> >> results. > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also please > > >> take a > > >> >> > > look? > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > Regards, > > >> >> > > > >>> > > > > > Saikat > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil > > >> Gkatziouras > > >> >> < > > >> >> > > > >>> > > > > > gkatzio...@gmail.com> > > >> >> > > > >>> > > > > > wrote: > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > > Hi @Saikat Maitra > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > I have my implementation on an Ignite fork on > > my > > >> >> > personal > > >> >> > > > >>> GitHub > > >> >> > > > >>> > > > > account > > >> >> > > > >>> > > > > > > [1] > > >> >> > > > >>> > > > > > > It is based on the Kafka implementation which > > is > > >> of > > >> >> a > > >> >> > > > polling > > >> >> > > > >>> > > nature. > > >> >> > > > >>> > > > > > > GCP does not provide a service like Kafka. > > >> Instead > > >> >> it > > >> >> > > > >>> provides us > > >> >> > > > >>> > > > with > > >> >> > > > >>> > > > > > > Pub/Sub which is just like Kafka. > > >> >> > > > >>> > > > > > > Therefore using the Kafka streamer as a > > guideline > > >> >> made > > >> >> > > more > > >> >> > > > >>> sense > > >> >> > > > >>> > > to > > >> >> > > > >>> > > > > me. > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > Apart from the implementation I have also > > added a > > >> >> unit > > >> >> > > > test. > > >> >> > > > >>> > > > > > > It definitely needs some polishing and a > double > > >> >> check > > >> >> > on > > >> >> > > my > > >> >> > > > >>> side > > >> >> > > > >>> > to > > >> >> > > > >>> > > > > make > > >> >> > > > >>> > > > > > > the review smooth. > > >> >> > > > >>> > > > > > > For the tests a local Pub/Sub [2] server has > to > > >> be > > >> >> > setup > > >> >> > > > >>> however > > >> >> > > > >>> > I > > >> >> > > > >>> > > am > > >> >> > > > >>> > > > > not > > >> >> > > > >>> > > > > > > sure which is the best way to do so with > > regards > > >> to > > >> >> > your > > >> >> > > > >>> CI/CD. > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > [1] > > >> >> > > > >>> > > > > >> >> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > >> >> > > > >>> > > > > > > [2] > > >> >> > > > >>> > https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > *Emmanouil Gkatziouras* > > >> >> > > > >>> > > > > > > https://egkatzioura.com/ | > > >> >> > > > >>> > > > > > > > > >> https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> > > > >>> > > > > > > https://github.com/gkatzioura > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > > >> >> > > > >>> > > saikat.mai...@gmail.com > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > > wrote: > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > > Hello Emmanouil, > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > Can you please share more info about the > > >> >> > > implementation? > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > We have similar implementation for data > > >> streamers > > >> >> > with > > >> >> > > > >>> Kafka > > >> >> > > > >>> > and > > >> >> > > > >>> > > > > > RocketMQ > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > > >> >> > https://apacheignite-mix.readme.io/docs/kafka-streamer > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > > >> >> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > Please review and share if you have any > > >> questions. > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > Regards, > > >> >> > > > >>> > > > > > > > Saikat > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil > > >> >> > Gkatziouras < > > >> >> > > > >>> > > > > > > > gkatzio...@gmail.com> > > >> >> > > > >>> > > > > > > > wrote: > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > Hi all, > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google > Cloud's > > >> >> Pub/Sub > > >> >> > > [1] > > >> >> > > > >>> > > > > > > > > Yes it is indeed cloud specific. It is > the > > >> >> > messaging > > >> >> > > > >>> service > > >> >> > > > >>> > > > > provided > > >> >> > > > >>> > > > > > > by > > >> >> > > > >>> > > > > > > > > GCP. > > >> >> > > > >>> > > > > > > > > I have made the integration with it, but > I > > >> >> haven't > > >> >> > > > >>> published > > >> >> > > > >>> > it > > >> >> > > > >>> > > > yet > > >> >> > > > >>> > > > > > (I > > >> >> > > > >>> > > > > > > am > > >> >> > > > >>> > > > > > > > > waiting for the discussion/ticket and > > proceed > > >> >> with > > >> >> > > the > > >> >> > > > >>> > > > > contribution). > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > Kind regards, > > >> >> > > > >>> > > > > > > > > Emmanouil > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > [1] > > >> >> https://cloud.google.com/pubsub/docs/overview > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > *Emmanouil Gkatziouras* > > >> >> > > > >>> > > > > > > > > https://egkatzioura.com/ | > > >> >> > > > >>> > > > > > > > > > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> > > > >>> > > > > > > > > https://github.com/gkatzioura > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis > Magda < > > >> >> > > > >>> dma...@apache.org > > >> >> > > > >>> > > > > >> >> > > > >>> > > > > wrote: > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > > Hello Emmanouil, > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > Are you referring to any specific > > >> >> implementation > > >> >> > > > >>> tailored > > >> >> > > > >>> > > for a > > >> >> > > > >>> > > > > > > cloud? > > >> >> > > > >>> > > > > > > > If > > >> >> > > > >>> > > > > > > > > > I'm not mistaken you were mentioning > > >> AWS/GCE. > > >> >> Is > > >> >> > > > there > > >> >> > > > >>> any > > >> >> > > > >>> > > > Github > > >> >> > > > >>> > > > > > > > source > > >> >> > > > >>> > > > > > > > > > code? > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > Here is a list of existing integrations > > >> with > > >> >> > > > streaming > > >> >> > > > >>> and > > >> >> > > > >>> > > > > pub/sub > > >> >> > > > >>> > > > > > > > > > frameworks: > > >> >> > > > >>> > > > > > > > > > > > >> >> https://apacheignite-mix.readme.io/docs/overview > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > with the source code in Ignite master: > > >> >> > > > >>> > > > > > > > > > > > >> >> > > https://github.com/apache/ignite/tree/master/modules > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > Check out those resources to see what > are > > >> the > > >> >> > most > > >> >> > > > >>> suitable > > >> >> > > > >>> > > > > Ignite > > >> >> > > > >>> > > > > > > APIs > > >> >> > > > >>> > > > > > > > > for > > >> >> > > > >>> > > > > > > > > > such integrations. > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > @Saikat Maitra < > saikat.mai...@gmail.com> > > , > > >> >> > @Roman > > >> >> > > < > > >> >> > > > >>> > > > > > rsht...@yahoo.com > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > , > > >> >> > > > >>> > > > > > > > > as > > >> >> > > > >>> > > > > > > > > > streaming technologies > > >> >> contributors/maintainers, > > >> >> > > > could > > >> >> > > > >>> you > > >> >> > > > >>> > > > please > > >> >> > > > >>> > > > > > > join > > >> >> > > > >>> > > > > > > > > this > > >> >> > > > >>> > > > > > > > > > thread? > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > - > > >> >> > > > >>> > > > > > > > > > Denis > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM > Emmanouil > > >> >> > > > Gkatziouras < > > >> >> > > > >>> > > > > > > > > > gkatzio...@gmail.com> > > >> >> > > > >>> > > > > > > > > > wrote: > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > Hello everyone, > > >> >> > > > >>> > > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > I started using Ignite lately. Part > of > > my > > >> >> work > > >> >> > > > >>> involves > > >> >> > > > >>> > > > > Pub/Sub, > > >> >> > > > >>> > > > > > > > thus I > > >> >> > > > >>> > > > > > > > > > > created a pub/sub streamer. > > >> >> > > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. > > >> >> > > > >>> > > > > > > > > > > Here's how the streamer works. Every > > node > > >> >> > creates > > >> >> > > > an > > >> >> > > > >>> > > instance > > >> >> > > > >>> > > > > of > > >> >> > > > >>> > > > > > a > > >> >> > > > >>> > > > > > > > > > Pub/Sub > > >> >> > > > >>> > > > > > > > > > > client. > > >> >> > > > >>> > > > > > > > > > > The clients subscribe to the same > topic > > >> >> using > > >> >> > the > > >> >> > > > >>> same > > >> >> > > > >>> > > > > > subscription > > >> >> > > > >>> > > > > > > > > name. > > >> >> > > > >>> > > > > > > > > > > A message is being sent and one of > the > > >> nodes > > >> >> > will > > >> >> > > > >>> receive > > >> >> > > > >>> > > the > > >> >> > > > >>> > > > > > > > message, > > >> >> > > > >>> > > > > > > > > > > process it and then forward it to > > Ignite > > >> >> using > > >> >> > > the > > >> >> > > > >>> > > extractor > > >> >> > > > >>> > > > > > > > provided. > > >> >> > > > >>> > > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > Any thoughts? > > >> >> > > > >>> > > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* > > >> >> > > > >>> > > > > > > > > > > https://egkatzioura.com/ | > > >> >> > > > >>> > > > > > > > > > > > > >> >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> >> > > > >>> > > > > > > > > > > https://github.com/gkatzioura > > >> >> > > > >>> > > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > > >> >> > > > >>> > > > > > > > > > > >> >> > > > >>> > > > > > > > > > >> >> > > > >>> > > > > > > > > >> >> > > > >>> > > > > > > > >> >> > > > >>> > > > > > > >> >> > > > >>> > > > > > >> >> > > > >>> > > > > >> >> > > > >>> > > > >> >> > > > >>> > > >> >> > > > >> > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > > >> > > > > > >
