Hello, As discussed I have created following discussion threads on our migration proposals for Apache Ignite extensions:
http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html I will share update as I hear more information. Regards, Saikat On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hello Denis, > > Sure, sounds good. I will create a separate discussion thread to get > community feedback on whether we > should join the Bahir or kick-off "Ignite Extensions" project. > > Regards, > Saikat > > On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dma...@apache.org> wrote: > >> Folks, >> >> The concept of Apache Bahir is precisely what we need! Saikat, thanks for >> doing the research. Why I believe the * idea* fits us well: >> >> - All integrations can be stored in separate Github repositories and >> have their dev lifecycles. We've not obliged to couple all the >> integrations >> in a single repo. For instance, Spark can be located in a dedicated >> repo >> while streaming integrations might be in a single one. It's up to us. >> - All the repositories and integrations belong to ASF. We're not >> dumping >> them on Github but rather keep supporting and developing in accordance >> with >> ASF vision and practices. >> - There is a way to reward contributors via committership and PMC >> membership. You won't get this if a project is just one of the >> millions of >> Github projects. >> >> Saikat, as Ignite PMC member, could you please kick-off a separate >> dev-list >> discussion to involve more community members and referring to this thread? >> I think the primary question the community needs to answer whether we >> should join the Bahir or kick-off "Ignite Extensions" project? >> >> - >> Denis >> >> >> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <zaleslaw....@gmail.com> >> wrote: >> >> > Maybe we could move all our Streaming Integrations there, but what is >> about >> > maintaining and committer permissions to the new repositories? >> > I see the list of the committers and PMC members there >> > https://bahir.apache.org/community-members/ >> > >> > Could somebody from Ignite community be added to this list as a >> > PMC/committer to maintain Ignite integrations? >> > If yes, I'd happy to join this project and collaborate with these guys >> > together >> > If no, and we should start from the zero level with external PRs - hmmm, >> > it's better to have our own external repository with ApacheBahirr >> ideology. >> > >> > Also, I agree, that all connectors could be moved there (in ApacheBahirr >> > like repository), but not all modules from the page >> > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > because >> > they use different parts of core modules and could be developed with >> > different velocity. >> > >> > In reality, before creation of new repositories we need wide discussion >> > based on the document mentioned above. >> > >> > P.S. Of course, we could experiment in the next release 2.8 with one or >> two >> > integrations like twitter or ZeroMQ >> > Also, I'd like Denis Magda idea for the separate repositories in Apache >> > Github to maintain them by anybody. >> > >> > >> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.mai...@gmail.com>: >> > >> > > Hi Denis, Emmanouil >> > > >> > > Thank you for your email. I think creating a separate integration >> project >> > > in github and also proposing it as an apache incubator project will >> be a >> > > good move. The other separate integration modules >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > can >> > > be moved to this new apache incubator project. >> > > >> > > There are similar integration available for Flink and Spark in Apache >> > Bahir >> > > https://bahir.apache.org/ >> > > >> > > Do you think we should reach out to Apache Bahir community with a >> > proposal >> > > or we should plan to create a separate Apache Incubator project? >> > > >> > > Regards, >> > > Saikat >> > > >> > > >> > > >> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < >> > > gkatzio...@gmail.com> >> > > wrote: >> > > >> > > > Hi all! >> > > > >> > > > Based on the discussions most probably this is going to be on >> another >> > > > GitHub repo. Therefore I will prepare a standalone project with the >> > > Pub/Sub >> > > > module and once the repository is created a pull request shall be >> > issued. >> > > > If there is anything else I can do in order to facilitate this >> please >> > let >> > > > me know. >> > > > >> > > > If a ticket is created for this issue, I shall be able to create a >> pull >> > > > request and thus a build will take place on ignite's CI. >> > > > If everyone is aligned with adding a Pub/Sub implementation may I >> > create >> > > a >> > > > ticket on JIRA? >> > > > >> > > > My next question has to do on how Ignite configures the build jobs >> in >> > > Team >> > > > City. The unit tests for my implementation need a local Pub/Sub >> server >> > up >> > > > and running (they provide one for testing purposes). Will this take >> > > effect >> > > > in a form of a build script inside my implementation or is it >> something >> > > > that should be configured on Team City? >> > > > >> > > > Kind regards, >> > > > Emmanouil >> > > > >> > > > >> > > > *Emmanouil Gkatziouras* >> > > > https://egkatzioura.com/ | >> > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > > https://github.com/gkatzioura >> > > > >> > > > >> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dma...@apache.org> >> wrote: >> > > > >> > > > > Emmanouil, Saikat, >> > > > > >> > > > > After contemplating on this, let me propose another way with this >> > > > > particular integration that is aligned with our modularization >> > > endeavor. >> > > > > The modularization [1] defines the Ignite core, Ignite modules, >> and >> > > > > independent integrations. The first two are to be located in >> Ignite >> > > > > repositories and to be tested/updated by the community for every >> > > release. >> > > > > The modules are a subset of existing integrations that are adopted >> > > widely >> > > > > and require community attention. >> > > > > >> > > > > As for the independent integrations [1], those are supposed to be >> > > located >> > > > > in a separate GitHub repo. I would advise us to start implementing >> > this >> > > > > concept with the Pub/Sub integration. Even though it will be >> located >> > > in a >> > > > > separate Github repo, it will be listed among officially available >> > > > > integrations on the Ignite website featuring Emmanouil as a >> > > contributor. >> > > > > >> > > > > What do you think? >> > > > > >> > > > > >> > > > > [1] >> > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >> > > > > >> > > > > >> > > > > >> > > > > - >> > > > > Denis >> > > > > >> > > > > >> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < >> > saikat.mai...@gmail.com >> > > > >> > > > > wrote: >> > > > > >> > > > > > Hi Emmanouil, >> > > > > > >> > > > > > The changes looks good to me. I wanted to check if you are also >> > able >> > > to >> > > > > > validate that the once data is added to Ignite cluster you are >> also >> > > > able >> > > > > to >> > > > > > access it using another Ignite client or using rest endpoints. >> > > > > > >> > > > > > We use Teamcity for CI process, the details are as follows >> > > > > > >> > > > > > Trigger validation of those test suites that have been affected >> by >> > > your >> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: < >> > > > > http://204.14.53.153/ >> > > > > > > >> > > > > > >> > > > > > - Locate a test suite you have to check, press button named >> > "..." >> > > > that >> > > > > > is located on the left of "Run" button. "Run custom build" >> > dialog >> > > > will >> > > > > > appear; >> > > > > > - Go to "Changes" tab and choose >> > "pull/<pull-request-number>/head" >> > > > in >> > > > > > "Build branch" dropdown list; >> > > > > > - Press "Run build" button and monitor tests results. >> > > > > > >> > > > > > >> > > > > > >> > > > > > @Roman rsht...@yahoo.com - can you also please take a look? >> > > > > > >> > > > > > Regards, >> > > > > > Saikat >> > > > > > >> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < >> > > > > > gkatzio...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi @Saikat Maitra >> > > > > > > >> > > > > > > I have my implementation on an Ignite fork on my personal >> GitHub >> > > > > account >> > > > > > > [1] >> > > > > > > It is based on the Kafka implementation which is of a polling >> > > nature. >> > > > > > > GCP does not provide a service like Kafka. Instead it >> provides us >> > > > with >> > > > > > > Pub/Sub which is just like Kafka. >> > > > > > > Therefore using the Kafka streamer as a guideline made more >> sense >> > > to >> > > > > me. >> > > > > > > >> > > > > > > Apart from the implementation I have also added a unit test. >> > > > > > > It definitely needs some polishing and a double check on my >> side >> > to >> > > > > make >> > > > > > > the review smooth. >> > > > > > > For the tests a local Pub/Sub [2] server has to be setup >> however >> > I >> > > am >> > > > > not >> > > > > > > sure which is the best way to do so with regards to your >> CI/CD. >> > > > > > > >> > > > > > > [1] >> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >> > > > > > > [2] >> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ >> > > > > > > >> > > > > > > *Emmanouil Gkatziouras* >> > > > > > > https://egkatzioura.com/ | >> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > > > > > https://github.com/gkatzioura >> > > > > > > >> > > > > > > >> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < >> > > saikat.mai...@gmail.com >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hello Emmanouil, >> > > > > > > > >> > > > > > > > Can you please share more info about the implementation? >> > > > > > > > >> > > > > > > > We have similar implementation for data streamers with Kafka >> > and >> > > > > > RocketMQ >> > > > > > > > >> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer >> > > > > > > > >> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer >> > > > > > > > >> > > > > > > > Please review and share if you have any questions. >> > > > > > > > >> > > > > > > > Regards, >> > > > > > > > Saikat >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < >> > > > > > > > gkatzio...@gmail.com> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi all, >> > > > > > > > > >> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] >> > > > > > > > > Yes it is indeed cloud specific. It is the messaging >> service >> > > > > provided >> > > > > > > by >> > > > > > > > > GCP. >> > > > > > > > > I have made the integration with it, but I haven't >> published >> > it >> > > > yet >> > > > > > (I >> > > > > > > am >> > > > > > > > > waiting for the discussion/ticket and proceed with the >> > > > > contribution). >> > > > > > > > > >> > > > > > > > > Kind regards, >> > > > > > > > > Emmanouil >> > > > > > > > > >> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview >> > > > > > > > > >> > > > > > > > > *Emmanouil Gkatziouras* >> > > > > > > > > https://egkatzioura.com/ | >> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > > > > > > > https://github.com/gkatzioura >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < >> dma...@apache.org >> > > >> > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hello Emmanouil, >> > > > > > > > > > >> > > > > > > > > > Are you referring to any specific implementation >> tailored >> > > for a >> > > > > > > cloud? >> > > > > > > > If >> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there >> any >> > > > Github >> > > > > > > > source >> > > > > > > > > > code? >> > > > > > > > > > >> > > > > > > > > > Here is a list of existing integrations with streaming >> and >> > > > > pub/sub >> > > > > > > > > > frameworks: >> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview >> > > > > > > > > > >> > > > > > > > > > with the source code in Ignite master: >> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules >> > > > > > > > > > >> > > > > > > > > > Check out those resources to see what are the most >> suitable >> > > > > Ignite >> > > > > > > APIs >> > > > > > > > > for >> > > > > > > > > > such integrations. >> > > > > > > > > > >> > > > > > > > > > @Saikat Maitra <saikat.mai...@gmail.com> , @Roman < >> > > > > > rsht...@yahoo.com >> > > > > > > > >> > > > > > > > , >> > > > > > > > > as >> > > > > > > > > > streaming technologies contributors/maintainers, could >> you >> > > > please >> > > > > > > join >> > > > > > > > > this >> > > > > > > > > > thread? >> > > > > > > > > > >> > > > > > > > > > - >> > > > > > > > > > Denis >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < >> > > > > > > > > > gkatzio...@gmail.com> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Hello everyone, >> > > > > > > > > > > >> > > > > > > > > > > I started using Ignite lately. Part of my work >> involves >> > > > > Pub/Sub, >> > > > > > > > thus I >> > > > > > > > > > > created a pub/sub streamer. >> > > > > > > > > > > Pub/Sub is pretty close to Kafka. >> > > > > > > > > > > Here's how the streamer works. Every node creates an >> > > instance >> > > > > of >> > > > > > a >> > > > > > > > > > Pub/Sub >> > > > > > > > > > > client. >> > > > > > > > > > > The clients subscribe to the same topic using the same >> > > > > > subscription >> > > > > > > > > name. >> > > > > > > > > > > A message is being sent and one of the nodes will >> receive >> > > the >> > > > > > > > message, >> > > > > > > > > > > process it and then forward it to Ignite using the >> > > extractor >> > > > > > > > provided. >> > > > > > > > > > > >> > > > > > > > > > > Any thoughts? >> > > > > > > > > > > >> > > > > > > > > > > *Emmanouil Gkatziouras* >> > > > > > > > > > > https://egkatzioura.com/ | >> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ >> > > > > > > > > > > https://github.com/gkatzioura >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
