Hi Denis, Emmanouil Thank you for your email. I think creating a separate integration project in github and also proposing it as an apache incubator project will be a good move. The other separate integration modules https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations can be moved to this new apache incubator project.
There are similar integration available for Flink and Spark in Apache Bahir https://bahir.apache.org/ Do you think we should reach out to Apache Bahir community with a proposal or we should plan to create a separate Apache Incubator project? Regards, Saikat On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <gkatzio...@gmail.com> wrote: > Hi all! > > Based on the discussions most probably this is going to be on another > GitHub repo. Therefore I will prepare a standalone project with the Pub/Sub > module and once the repository is created a pull request shall be issued. > If there is anything else I can do in order to facilitate this please let > me know. > > If a ticket is created for this issue, I shall be able to create a pull > request and thus a build will take place on ignite's CI. > If everyone is aligned with adding a Pub/Sub implementation may I create a > ticket on JIRA? > > My next question has to do on how Ignite configures the build jobs in Team > City. The unit tests for my implementation need a local Pub/Sub server up > and running (they provide one for testing purposes). Will this take effect > in a form of a build script inside my implementation or is it something > that should be configured on Team City? > > Kind regards, > Emmanouil > > > *Emmanouil Gkatziouras* > https://egkatzioura.com/ | > https://www.linkedin.com/in/gkatziourasemmanouil/ > https://github.com/gkatzioura > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> wrote: > > > Emmanouil, Saikat, > > > > After contemplating on this, let me propose another way with this > > particular integration that is aligned with our modularization endeavor. > > The modularization [1] defines the Ignite core, Ignite modules, and > > independent integrations. The first two are to be located in Ignite > > repositories and to be tested/updated by the community for every release. > > The modules are a subset of existing integrations that are adopted widely > > and require community attention. > > > > As for the independent integrations [1], those are supposed to be located > > in a separate GitHub repo. I would advise us to start implementing this > > concept with the Pub/Sub integration. Even though it will be located in a > > separate Github repo, it will be listed among officially available > > integrations on the Ignite website featuring Emmanouil as a contributor. > > > > What do you think? > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations > > > > > > > > - > > Denis > > > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <saikat.mai...@gmail.com> > > wrote: > > > > > Hi Emmanouil, > > > > > > The changes looks good to me. I wanted to check if you are also able to > > > validate that the once data is added to Ignite cluster you are also > able > > to > > > access it using another Ignite client or using rest endpoints. > > > > > > We use Teamcity for CI process, the details are as follows > > > > > > Trigger validation of those test suites that have been affected by your > > > changes on TeamCity <http://ci.ignite.apache.org/>: < > > http://204.14.53.153/ > > > > > > > > > > - Locate a test suite you have to check, press button named "..." > that > > > is located on the left of "Run" button. "Run custom build" dialog > will > > > appear; > > > - Go to "Changes" tab and choose "pull/<pull-request-number>/head" > in > > > "Build branch" dropdown list; > > > - Press "Run build" button and monitor tests results. > > > > > > > > > > > > @Roman rsht...@yahoo.com - can you also please take a look? > > > > > > Regards, > > > Saikat > > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > > > gkatzio...@gmail.com> > > > wrote: > > > > > > > Hi @Saikat Maitra > > > > > > > > I have my implementation on an Ignite fork on my personal GitHub > > account > > > > [1] > > > > It is based on the Kafka implementation which is of a polling nature. > > > > GCP does not provide a service like Kafka. Instead it provides us > with > > > > Pub/Sub which is just like Kafka. > > > > Therefore using the Kafka streamer as a guideline made more sense to > > me. > > > > > > > > Apart from the implementation I have also added a unit test. > > > > It definitely needs some polishing and a double check on my side to > > make > > > > the review smooth. > > > > For the tests a local Pub/Sub [2] server has to be setup however I am > > not > > > > sure which is the best way to do so with regards to your CI/CD. > > > > > > > > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ > > > > > > > > *Emmanouil Gkatziouras* > > > > https://egkatzioura.com/ | > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > https://github.com/gkatzioura > > > > > > > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <saikat.mai...@gmail.com > > > > > > wrote: > > > > > > > > > Hello Emmanouil, > > > > > > > > > > Can you please share more info about the implementation? > > > > > > > > > > We have similar implementation for data streamers with Kafka and > > > RocketMQ > > > > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > > > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > > > > > > > > > Please review and share if you have any questions. > > > > > > > > > > Regards, > > > > > Saikat > > > > > > > > > > > > > > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > > > > > gkatzio...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] > > > > > > Yes it is indeed cloud specific. It is the messaging service > > provided > > > > by > > > > > > GCP. > > > > > > I have made the integration with it, but I haven't published it > yet > > > (I > > > > am > > > > > > waiting for the discussion/ticket and proceed with the > > contribution). > > > > > > > > > > > > Kind regards, > > > > > > Emmanouil > > > > > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > https://egkatzioura.com/ | > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <dma...@apache.org> > > wrote: > > > > > > > > > > > > > Hello Emmanouil, > > > > > > > > > > > > > > Are you referring to any specific implementation tailored for a > > > > cloud? > > > > > If > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any > Github > > > > > source > > > > > > > code? > > > > > > > > > > > > > > Here is a list of existing integrations with streaming and > > pub/sub > > > > > > > frameworks: > > > > > > > https://apacheignite-mix.readme.io/docs/overview > > > > > > > > > > > > > > with the source code in Ignite master: > > > > > > > https://github.com/apache/ignite/tree/master/modules > > > > > > > > > > > > > > Check out those resources to see what are the most suitable > > Ignite > > > > APIs > > > > > > for > > > > > > > such integrations. > > > > > > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman < > > > rsht...@yahoo.com > > > > > > > > > > , > > > > > > as > > > > > > > streaming technologies contributors/maintainers, could you > please > > > > join > > > > > > this > > > > > > > thread? > > > > > > > > > > > > > > - > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < > > > > > > > gkatzio...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > I started using Ignite lately. Part of my work involves > > Pub/Sub, > > > > > thus I > > > > > > > > created a pub/sub streamer. > > > > > > > > Pub/Sub is pretty close to Kafka. > > > > > > > > Here's how the streamer works. Every node creates an instance > > of > > > a > > > > > > > Pub/Sub > > > > > > > > client. > > > > > > > > The clients subscribe to the same topic using the same > > > subscription > > > > > > name. > > > > > > > > A message is being sent and one of the nodes will receive the > > > > > message, > > > > > > > > process it and then forward it to Ignite using the extractor > > > > > provided. > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > > https://egkatzioura.com/ | > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
