Hi Emmanouil, The latest build on teamcity has passed on Flink pull request.
https://ci.ignite.apache.org/viewLog.html?buildId=4788928&buildTypeId=IgniteExtensions_Build&tab=buildResultsDiv&branch_IgniteExtensions=pull%2F1%2Fhead you should be able to take the changes I made in my PR and run build on your pull request. Please let me know if you have any questions. Regards, Saikat On Sun, Nov 24, 2019 at 10:44 AM Saikat Maitra <saikat.mai...@gmail.com> wrote: > Hi Emmanouil, > > I have fixed the Flink module testsuites and have setup new project in > teamcity. > > > https://ci.ignite.apache.org/project.html?projectId=IgniteExtensions&tab=projectOverview > > I am looking into setting up the build. > > Regards, > Saikat > > On Sat, Nov 23, 2019 at 9:52 AM Saikat Maitra <saikat.mai...@gmail.com> > wrote: > >> Hi Emmanouil, >> >> Thank you for your email. Yes, the plan is once the Flink PR is merged it >> will provide the Licence, parent POMs etc and you can rebase from master >> branch and apply the changes on top of it. >> >> I am getting some issues with test failures in local with >> GridTestProperties as the test.properties is not present in this new >> project but should be available from dependencies. >> >> Once I address this issue, I will go ahead and merge the changes and then >> we can take it from there. >> >> Regards, >> Saikat >> >> On Fri, Nov 22, 2019 at 5:23 PM Emmanouil Gkatziouras < >> gkatzio...@gmail.com> wrote: >> >>> Hi all, >>> >>> I made my first pull request [1]. Since this project is brand new (no >>> parent poms, licensing), I reckoned it was best to use Saikat's branch on >>> Flink. >>> I suppose the Flink branch will be merged first. If not please give me >>> guidelines on how I should proceed next. >>> >>> Kind regards >>> Emmanouil >>> >>> [1] https://github.com/apache/ignite-extensions/pull/2 >>> >>> *Emmanouil Gkatziouras* >>> https://egkatzioura.com/ | >>> https://www.linkedin.com/in/gkatziourasemmanouil/ >>> https://github.com/gkatzioura >>> >>> >>> On Fri, 22 Nov 2019 at 20:55, Denis Magda <dma...@apache.org> wrote: >>> >>> > Awesome, ping us whenever you're ready! >>> > >>> > - >>> > Denis >>> > >>> > >>> > On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras < >>> > gkatzio...@gmail.com> >>> > wrote: >>> > >>> > > Hi all! >>> > > >>> > > I am sorry for being late on that. I was trying to refactor the test >>> in >>> > > order not to be in need of any external tools or spinning up a >>> server. >>> > > I did forked the new repo and indeed my changes there, so a pull >>> request >>> > is >>> > > a matter of time! >>> > > >>> > > Kind regards >>> > > >>> > > *Emmanouil Gkatziouras* >>> > > https://egkatzioura.com/ | >>> > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > https://github.com/gkatzioura >>> > > >>> > > >>> > > On Fri, 22 Nov 2019 at 20:45, Denis Magda <dma...@apache.org> wrote: >>> > > >>> > > > Hi Emmanouil, >>> > > > >>> > > > Do you have any questions or need any support from the community? >>> > > > >>> > > > - >>> > > > Denis >>> > > > >>> > > > >>> > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra < >>> saikat.mai...@gmail.com >>> > > >>> > > > wrote: >>> > > > >>> > > >> Hi Emmanouil, >>> > > >> >>> > > >> Can you please take a looks at dev utils, if this is something >>> you are >>> > > >> looking for >>> > > >> https://github.com/apache/ignite/tree/master/modules/dev-utils? >>> > > >> >>> > > >> IMO, if you can release Pub/Sub server in maven and then use it as >>> > > >> dependency, that would be great. >>> > > >> >>> > > >> Regards, >>> > > >> Saikat >>> > > >> >>> > > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra < >>> > saikat.mai...@gmail.com> >>> > > >> wrote: >>> > > >> >>> > > >> > Hi Emmanouil, >>> > > >> > >>> > > >> > The master branch has been initialized. I have raised a PR to >>> > migrate >>> > > >> > Flink module to ignite-extensions. >>> > > >> > >>> > > >> > PR https://github.com/apache/ignite-extensions/pull/1 >>> > > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 >>> > > >> > >>> > > >> > Regards, >>> > > >> > Saikat >>> > > >> > >>> > > >> > >>> > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras < >>> > > >> > gkatzio...@gmail.com> wrote: >>> > > >> > >>> > > >> >> Hi all! >>> > > >> >> >>> > > >> >> This is an update. The integration is tested with mocks based >>> on >>> > the >>> > > >> >> original Ignite repo [1]. >>> > > >> >> I am currently waiting for the first commits on the new >>> integration >>> > > >> >> project >>> > > >> >> to take place in order to create my pull request accordingly >>> to the >>> > > >> >> initial >>> > > >> >> commits. >>> > > >> >> In order to simulate the Pub/Sub server in a test I used a test >>> > > server >>> > > >> >> implementation [2]. Unfortunately they don't have this util >>> > uploaded >>> > > >> in a >>> > > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache >>> > > License >>> > > >> >> 2.0). Is there a repo for utilities that Ignite uses for the >>> > > >> test/build? >>> > > >> >> I have also created a ticket [3] >>> > > >> >> >>> > > >> >> Kind regards >>> > > >> >> >>> > > >> >> [1] >>> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >>> > > >> >> [2] >>> https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator >>> > > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262 >>> > > >> >> >>> > > >> >> *Emmanouil Gkatziouras* >>> > > >> >> https://egkatzioura.com/ | >>> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > >> >> https://github.com/gkatzioura >>> > > >> >> >>> > > >> >> >>> > > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra < >>> > saikat.mai...@gmail.com >>> > > > >>> > > >> >> wrote: >>> > > >> >> >>> > > >> >> > Hello Ilya, >>> > > >> >> > >>> > > >> >> > Thank you for your email. Yes, before we remove any >>> integration >>> > we >>> > > >> will >>> > > >> >> > move them to another repository and release. >>> > > >> >> > >>> > > >> >> > We are thinking we will start with Pub/Sub Streamer >>> > implementation >>> > > >> and >>> > > >> >> > continue to move other integration as mentioned in the >>> following >>> > > doc. >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > > >> >> >>> > > >> >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > >> >> > >>> > > >> >> > Regards, >>> > > >> >> > Saikat >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev < >>> > > >> >> > ilya.kasnach...@gmail.com> >>> > > >> >> > wrote: >>> > > >> >> > >>> > > >> >> > > Hello! >>> > > >> >> > > >>> > > >> >> > > My take on this: we want to offload some of integrations, >>> but >>> > > >> please >>> > > >> >> > commit >>> > > >> >> > > and release them to another project first, and only then we >>> > will >>> > > >> >> decide >>> > > >> >> > to >>> > > >> >> > > remove them from our repository. >>> > > >> >> > > >>> > > >> >> > > Please also make sure they are available with same artifact >>> > name >>> > > >> and >>> > > >> >> API. >>> > > >> >> > > If this has to be changed, let's wait for AI 3.0 >>> > > >> >> > > >>> > > >> >> > > Regards, >>> > > >> >> > > -- >>> > > >> >> > > Ilya Kasnacheev >>> > > >> >> > > >>> > > >> >> > > >>> > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra < >>> > > >> saikat.mai...@gmail.com>: >>> > > >> >> > > >>> > > >> >> > > > Hello Emmanouil, >>> > > >> >> > > > >>> > > >> >> > > > As discussed earlier, I discussed with Apache Bahir >>> community >>> > > and >>> > > >> >> they >>> > > >> >> > > are >>> > > >> >> > > > interested to have Apache Ignite extensions as part of >>> Apache >>> > > >> Bahir >>> > > >> >> > > > project. >>> > > >> >> > > > >>> > > >> >> > > > Can you please share Pub/Sub streamer implementation >>> details >>> > > with >>> > > >> >> > Apache >>> > > >> >> > > > Bahir community and request for feedback. >>> > > >> >> > > > >>> > > >> >> > > > https://bahir.apache.org/ >>> > > >> >> > > > >>> > > >> >> > > > I have also requested for contributor access in Jira for >>> > Apache >>> > > >> >> Bahir >>> > > >> >> > > > project so that I can create issues and assign to myself. >>> > > >> >> > > > >>> > > >> >> > > > I can help with code reviews as well. >>> > > >> >> > > > >>> > > >> >> > > > Here is the email thread for reference >>> > > >> >> > > > >>> > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html >>> > > >> >> > > > >>> > > >> >> > > > Regards, >>> > > >> >> > > > Saikat >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > >>> > > >> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra < >>> > > >> >> saikat.mai...@gmail.com >>> > > >> >> > > >>> > > >> >> > > > wrote: >>> > > >> >> > > > >>> > > >> >> > > > > Hello, >>> > > >> >> > > > > >>> > > >> >> > > > > As discussed I have created following discussion >>> threads on >>> > > our >>> > > >> >> > > migration >>> > > >> >> > > > > proposals for Apache Ignite extensions: >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> > > > >>> > > >> >> > > >>> > > >> >> > >>> > > >> >> >>> > > >> >>> > > >>> > >>> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html >>> > > >> >> > > > > >>> > > >> >> > > > > I will share update as I hear more information. >>> > > >> >> > > > > >>> > > >> >> > > > > Regards, >>> > > >> >> > > > > Saikat >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> > > > > >>> > > >> >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra < >>> > > >> >> > saikat.mai...@gmail.com >>> > > >> >> > > > >>> > > >> >> > > > > wrote: >>> > > >> >> > > > > >>> > > >> >> > > > >> Hello Denis, >>> > > >> >> > > > >> >>> > > >> >> > > > >> Sure, sounds good. I will create a separate discussion >>> > > thread >>> > > >> to >>> > > >> >> get >>> > > >> >> > > > >> community feedback on whether we >>> > > >> >> > > > >> should join the Bahir or kick-off "Ignite Extensions" >>> > > project. >>> > > >> >> > > > >> >>> > > >> >> > > > >> Regards, >>> > > >> >> > > > >> Saikat >>> > > >> >> > > > >> >>> > > >> >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda < >>> > > >> dma...@apache.org> >>> > > >> >> > > wrote: >>> > > >> >> > > > >> >>> > > >> >> > > > >>> Folks, >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> The concept of Apache Bahir is precisely what we >>> need! >>> > > >> Saikat, >>> > > >> >> > thanks >>> > > >> >> > > > for >>> > > >> >> > > > >>> doing the research. Why I believe the * idea* fits us >>> > well: >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> - All integrations can be stored in separate >>> Github >>> > > >> >> repositories >>> > > >> >> > > and >>> > > >> >> > > > >>> have their dev lifecycles. We've not obliged to >>> couple >>> > > all >>> > > >> >> the >>> > > >> >> > > > >>> integrations >>> > > >> >> > > > >>> in a single repo. For instance, Spark can be >>> located >>> > in >>> > > a >>> > > >> >> > > dedicated >>> > > >> >> > > > >>> repo >>> > > >> >> > > > >>> while streaming integrations might be in a single >>> one. >>> > > >> It's >>> > > >> >> up >>> > > >> >> > to >>> > > >> >> > > > us. >>> > > >> >> > > > >>> - All the repositories and integrations belong to >>> ASF. >>> > > >> We're >>> > > >> >> not >>> > > >> >> > > > >>> dumping >>> > > >> >> > > > >>> them on Github but rather keep supporting and >>> > developing >>> > > >> in >>> > > >> >> > > > >>> accordance with >>> > > >> >> > > > >>> ASF vision and practices. >>> > > >> >> > > > >>> - There is a way to reward contributors via >>> > > committership >>> > > >> and >>> > > >> >> > PMC >>> > > >> >> > > > >>> membership. You won't get this if a project is >>> just >>> > one >>> > > of >>> > > >> >> the >>> > > >> >> > > > >>> millions of >>> > > >> >> > > > >>> Github projects. >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> Saikat, as Ignite PMC member, could you please >>> kick-off a >>> > > >> >> separate >>> > > >> >> > > > >>> dev-list >>> > > >> >> > > > >>> discussion to involve more community members and >>> > referring >>> > > to >>> > > >> >> this >>> > > >> >> > > > >>> thread? >>> > > >> >> > > > >>> I think the primary question the community needs to >>> > answer >>> > > >> >> whether >>> > > >> >> > we >>> > > >> >> > > > >>> should join the Bahir or kick-off "Ignite Extensions" >>> > > >> project? >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> - >>> > > >> >> > > > >>> Denis >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev < >>> > > >> >> > > > zaleslaw....@gmail.com> >>> > > >> >> > > > >>> wrote: >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> > Maybe we could move all our Streaming Integrations >>> > there, >>> > > >> but >>> > > >> >> > what >>> > > >> >> > > is >>> > > >> >> > > > >>> about >>> > > >> >> > > > >>> > maintaining and committer permissions to the new >>> > > >> repositories? >>> > > >> >> > > > >>> > I see the list of the committers and PMC members >>> there >>> > > >> >> > > > >>> > https://bahir.apache.org/community-members/ >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > Could somebody from Ignite community be added to >>> this >>> > > list >>> > > >> as >>> > > >> >> a >>> > > >> >> > > > >>> > PMC/committer to maintain Ignite integrations? >>> > > >> >> > > > >>> > If yes, I'd happy to join this project and >>> collaborate >>> > > with >>> > > >> >> these >>> > > >> >> > > > guys >>> > > >> >> > > > >>> > together >>> > > >> >> > > > >>> > If no, and we should start from the zero level with >>> > > >> external >>> > > >> >> PRs >>> > > >> >> > - >>> > > >> >> > > > >>> hmmm, >>> > > >> >> > > > >>> > it's better to have our own external repository >>> with >>> > > >> >> ApacheBahirr >>> > > >> >> > > > >>> ideology. >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > Also, I agree, that all connectors could be moved >>> there >>> > > (in >>> > > >> >> > > > >>> ApacheBahirr >>> > > >> >> > > > >>> > like repository), but not all modules from the page >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> > > >> >> > > >>> > > >> >> > >>> > > >> >> >>> > > >> >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > >> >> > > > >>> > because >>> > > >> >> > > > >>> > they use different parts of core modules and could >>> be >>> > > >> >> developed >>> > > >> >> > > with >>> > > >> >> > > > >>> > different velocity. >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > In reality, before creation of new repositories we >>> need >>> > > >> wide >>> > > >> >> > > > discussion >>> > > >> >> > > > >>> > based on the document mentioned above. >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > P.S. Of course, we could experiment in the next >>> release >>> > > 2.8 >>> > > >> >> with >>> > > >> >> > > one >>> > > >> >> > > > >>> or two >>> > > >> >> > > > >>> > integrations like twitter or ZeroMQ >>> > > >> >> > > > >>> > Also, I'd like Denis Magda idea for the separate >>> > > >> repositories >>> > > >> >> in >>> > > >> >> > > > Apache >>> > > >> >> > > > >>> > Github to maintain them by anybody. >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra < >>> > > >> >> > > saikat.mai...@gmail.com >>> > > >> >> > > > >: >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> > > Hi Denis, Emmanouil >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > Thank you for your email. I think creating a >>> separate >>> > > >> >> > integration >>> > > >> >> > > > >>> project >>> > > >> >> > > > >>> > > in github and also proposing it as an apache >>> > incubator >>> > > >> >> project >>> > > >> >> > > will >>> > > >> >> > > > >>> be a >>> > > >> >> > > > >>> > > good move. The other separate integration modules >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> > > >> >> > > >>> > > >> >> > >>> > > >> >> >>> > > >> >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > >> >> > > > >>> > > can >>> > > >> >> > > > >>> > > be moved to this new apache incubator project. >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > There are similar integration available for >>> Flink and >>> > > >> Spark >>> > > >> >> in >>> > > >> >> > > > Apache >>> > > >> >> > > > >>> > Bahir >>> > > >> >> > > > >>> > > https://bahir.apache.org/ >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > Do you think we should reach out to Apache Bahir >>> > > >> community >>> > > >> >> > with a >>> > > >> >> > > > >>> > proposal >>> > > >> >> > > > >>> > > or we should plan to create a separate Apache >>> > Incubator >>> > > >> >> > project? >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > Regards, >>> > > >> >> > > > >>> > > Saikat >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil >>> > Gkatziouras < >>> > > >> >> > > > >>> > > gkatzio...@gmail.com> >>> > > >> >> > > > >>> > > wrote: >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > > Hi all! >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > Based on the discussions most probably this is >>> > going >>> > > to >>> > > >> >> be on >>> > > >> >> > > > >>> another >>> > > >> >> > > > >>> > > > GitHub repo. Therefore I will prepare a >>> standalone >>> > > >> project >>> > > >> >> > with >>> > > >> >> > > > the >>> > > >> >> > > > >>> > > Pub/Sub >>> > > >> >> > > > >>> > > > module and once the repository is created a >>> pull >>> > > >> request >>> > > >> >> > shall >>> > > >> >> > > be >>> > > >> >> > > > >>> > issued. >>> > > >> >> > > > >>> > > > If there is anything else I can do in order to >>> > > >> facilitate >>> > > >> >> > this >>> > > >> >> > > > >>> please >>> > > >> >> > > > >>> > let >>> > > >> >> > > > >>> > > > me know. >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > If a ticket is created for this issue, I shall >>> be >>> > > able >>> > > >> to >>> > > >> >> > > create >>> > > >> >> > > > a >>> > > >> >> > > > >>> pull >>> > > >> >> > > > >>> > > > request and thus a build will take place on >>> > ignite's >>> > > >> CI. >>> > > >> >> > > > >>> > > > If everyone is aligned with adding a Pub/Sub >>> > > >> >> implementation >>> > > >> >> > > may I >>> > > >> >> > > > >>> > create >>> > > >> >> > > > >>> > > a >>> > > >> >> > > > >>> > > > ticket on JIRA? >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > My next question has to do on how Ignite >>> configures >>> > > the >>> > > >> >> build >>> > > >> >> > > > jobs >>> > > >> >> > > > >>> in >>> > > >> >> > > > >>> > > Team >>> > > >> >> > > > >>> > > > City. The unit tests for my implementation >>> need a >>> > > local >>> > > >> >> > Pub/Sub >>> > > >> >> > > > >>> server >>> > > >> >> > > > >>> > up >>> > > >> >> > > > >>> > > > and running (they provide one for testing >>> > purposes). >>> > > >> Will >>> > > >> >> > this >>> > > >> >> > > > take >>> > > >> >> > > > >>> > > effect >>> > > >> >> > > > >>> > > > in a form of a build script inside my >>> > implementation >>> > > or >>> > > >> >> is it >>> > > >> >> > > > >>> something >>> > > >> >> > > > >>> > > > that should be configured on Team City? >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > Kind regards, >>> > > >> >> > > > >>> > > > Emmanouil >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > *Emmanouil Gkatziouras* >>> > > >> >> > > > >>> > > > https://egkatzioura.com/ | >>> > > >> >> > > > >>> > > > >>> https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > >> >> > > > >>> > > > https://github.com/gkatzioura >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda < >>> > > >> >> dma...@apache.org >>> > > >> >> > > >>> > > >> >> > > > >>> wrote: >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > > Emmanouil, Saikat, >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > After contemplating on this, let me propose >>> > another >>> > > >> way >>> > > >> >> > with >>> > > >> >> > > > this >>> > > >> >> > > > >>> > > > > particular integration that is aligned with >>> our >>> > > >> >> > > modularization >>> > > >> >> > > > >>> > > endeavor. >>> > > >> >> > > > >>> > > > > The modularization [1] defines the Ignite >>> core, >>> > > >> Ignite >>> > > >> >> > > modules, >>> > > >> >> > > > >>> and >>> > > >> >> > > > >>> > > > > independent integrations. The first two are >>> to be >>> > > >> >> located >>> > > >> >> > in >>> > > >> >> > > > >>> Ignite >>> > > >> >> > > > >>> > > > > repositories and to be tested/updated by the >>> > > >> community >>> > > >> >> for >>> > > >> >> > > > every >>> > > >> >> > > > >>> > > release. >>> > > >> >> > > > >>> > > > > The modules are a subset of existing >>> integrations >>> > > >> that >>> > > >> >> are >>> > > >> >> > > > >>> adopted >>> > > >> >> > > > >>> > > widely >>> > > >> >> > > > >>> > > > > and require community attention. >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > As for the independent integrations [1], >>> those >>> > are >>> > > >> >> supposed >>> > > >> >> > > to >>> > > >> >> > > > be >>> > > >> >> > > > >>> > > located >>> > > >> >> > > > >>> > > > > in a separate GitHub repo. I would advise us >>> to >>> > > start >>> > > >> >> > > > >>> implementing >>> > > >> >> > > > >>> > this >>> > > >> >> > > > >>> > > > > concept with the Pub/Sub integration. Even >>> though >>> > > it >>> > > >> >> will >>> > > >> >> > be >>> > > >> >> > > > >>> located >>> > > >> >> > > > >>> > > in a >>> > > >> >> > > > >>> > > > > separate Github repo, it will be listed among >>> > > >> officially >>> > > >> >> > > > >>> available >>> > > >> >> > > > >>> > > > > integrations on the Ignite website featuring >>> > > >> Emmanouil >>> > > >> >> as a >>> > > >> >> > > > >>> > > contributor. >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > What do you think? >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > [1] >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> >>> > > >> >> > > > >>> > > >> >> > > >>> > > >> >> > >>> > > >> >> >>> > > >> >>> > > >>> > >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > - >>> > > >> >> > > > >>> > > > > Denis >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat >>> Maitra < >>> > > >> >> > > > >>> > saikat.mai...@gmail.com >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > > > wrote: >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > > Hi Emmanouil, >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > The changes looks good to me. I wanted to >>> check >>> > > if >>> > > >> you >>> > > >> >> > are >>> > > >> >> > > > also >>> > > >> >> > > > >>> > able >>> > > >> >> > > > >>> > > to >>> > > >> >> > > > >>> > > > > > validate that the once data is added to >>> Ignite >>> > > >> cluster >>> > > >> >> > you >>> > > >> >> > > > are >>> > > >> >> > > > >>> also >>> > > >> >> > > > >>> > > > able >>> > > >> >> > > > >>> > > > > to >>> > > >> >> > > > >>> > > > > > access it using another Ignite client or >>> using >>> > > rest >>> > > >> >> > > > endpoints. >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > We use Teamcity for CI process, the >>> details are >>> > > as >>> > > >> >> > follows >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > Trigger validation of those test suites >>> that >>> > have >>> > > >> been >>> > > >> >> > > > >>> affected by >>> > > >> >> > > > >>> > > your >>> > > >> >> > > > >>> > > > > > changes on TeamCity < >>> > > http://ci.ignite.apache.org/>: >>> > > >> < >>> > > >> >> > > > >>> > > > > http://204.14.53.153/ >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > - Locate a test suite you have to check, >>> > press >>> > > >> >> button >>> > > >> >> > > > named >>> > > >> >> > > > >>> > "..." >>> > > >> >> > > > >>> > > > that >>> > > >> >> > > > >>> > > > > > is located on the left of "Run" button. >>> "Run >>> > > >> custom >>> > > >> >> > > build" >>> > > >> >> > > > >>> > dialog >>> > > >> >> > > > >>> > > > will >>> > > >> >> > > > >>> > > > > > appear; >>> > > >> >> > > > >>> > > > > > - Go to "Changes" tab and choose >>> > > >> >> > > > >>> > "pull/<pull-request-number>/head" >>> > > >> >> > > > >>> > > > in >>> > > >> >> > > > >>> > > > > > "Build branch" dropdown list; >>> > > >> >> > > > >>> > > > > > - Press "Run build" button and monitor >>> tests >>> > > >> >> results. >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > @Roman rsht...@yahoo.com - can you also >>> please >>> > > >> take a >>> > > >> >> > > look? >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > Regards, >>> > > >> >> > > > >>> > > > > > Saikat >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil >>> > > >> Gkatziouras >>> > > >> >> < >>> > > >> >> > > > >>> > > > > > gkatzio...@gmail.com> >>> > > >> >> > > > >>> > > > > > wrote: >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > > > Hi @Saikat Maitra >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > I have my implementation on an Ignite >>> fork on >>> > > my >>> > > >> >> > personal >>> > > >> >> > > > >>> GitHub >>> > > >> >> > > > >>> > > > > account >>> > > >> >> > > > >>> > > > > > > [1] >>> > > >> >> > > > >>> > > > > > > It is based on the Kafka implementation >>> which >>> > > is >>> > > >> of >>> > > >> >> a >>> > > >> >> > > > polling >>> > > >> >> > > > >>> > > nature. >>> > > >> >> > > > >>> > > > > > > GCP does not provide a service like >>> Kafka. >>> > > >> Instead >>> > > >> >> it >>> > > >> >> > > > >>> provides us >>> > > >> >> > > > >>> > > > with >>> > > >> >> > > > >>> > > > > > > Pub/Sub which is just like Kafka. >>> > > >> >> > > > >>> > > > > > > Therefore using the Kafka streamer as a >>> > > guideline >>> > > >> >> made >>> > > >> >> > > more >>> > > >> >> > > > >>> sense >>> > > >> >> > > > >>> > > to >>> > > >> >> > > > >>> > > > > me. >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > Apart from the implementation I have also >>> > > added a >>> > > >> >> unit >>> > > >> >> > > > test. >>> > > >> >> > > > >>> > > > > > > It definitely needs some polishing and a >>> > double >>> > > >> >> check >>> > > >> >> > on >>> > > >> >> > > my >>> > > >> >> > > > >>> side >>> > > >> >> > > > >>> > to >>> > > >> >> > > > >>> > > > > make >>> > > >> >> > > > >>> > > > > > > the review smooth. >>> > > >> >> > > > >>> > > > > > > For the tests a local Pub/Sub [2] server >>> has >>> > to >>> > > >> be >>> > > >> >> > setup >>> > > >> >> > > > >>> however >>> > > >> >> > > > >>> > I >>> > > >> >> > > > >>> > > am >>> > > >> >> > > > >>> > > > > not >>> > > >> >> > > > >>> > > > > > > sure which is the best way to do so with >>> > > regards >>> > > >> to >>> > > >> >> > your >>> > > >> >> > > > >>> CI/CD. >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > [1] >>> > > >> >> > > > >>> > > >>> > > >> >> > >>> https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub >>> > > >> >> > > > >>> > > > > > > [2] >>> > > >> >> > > > >>> >>> > https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/ >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > *Emmanouil Gkatziouras* >>> > > >> >> > > > >>> > > > > > > https://egkatzioura.com/ | >>> > > >> >> > > > >>> > > > > > > >>> > > >> https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > >> >> > > > >>> > > > > > > https://github.com/gkatzioura >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat >>> Maitra < >>> > > >> >> > > > >>> > > saikat.mai...@gmail.com >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > > > > wrote: >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > > > Hello Emmanouil, >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > Can you please share more info about >>> the >>> > > >> >> > > implementation? >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > We have similar implementation for data >>> > > >> streamers >>> > > >> >> > with >>> > > >> >> > > > >>> Kafka >>> > > >> >> > > > >>> > and >>> > > >> >> > > > >>> > > > > > RocketMQ >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > https://apacheignite-mix.readme.io/docs/kafka-streamer >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> https://apacheignite-mix.readme.io/docs/rocketmq-streamer >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > Please review and share if you have any >>> > > >> questions. >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > Regards, >>> > > >> >> > > > >>> > > > > > > > Saikat >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM >>> Emmanouil >>> > > >> >> > Gkatziouras < >>> > > >> >> > > > >>> > > > > > > > gkatzio...@gmail.com> >>> > > >> >> > > > >>> > > > > > > > wrote: >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > Hi all, >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google >>> > Cloud's >>> > > >> >> Pub/Sub >>> > > >> >> > > [1] >>> > > >> >> > > > >>> > > > > > > > > Yes it is indeed cloud specific. It >>> is >>> > the >>> > > >> >> > messaging >>> > > >> >> > > > >>> service >>> > > >> >> > > > >>> > > > > provided >>> > > >> >> > > > >>> > > > > > > by >>> > > >> >> > > > >>> > > > > > > > > GCP. >>> > > >> >> > > > >>> > > > > > > > > I have made the integration with it, >>> but >>> > I >>> > > >> >> haven't >>> > > >> >> > > > >>> published >>> > > >> >> > > > >>> > it >>> > > >> >> > > > >>> > > > yet >>> > > >> >> > > > >>> > > > > > (I >>> > > >> >> > > > >>> > > > > > > am >>> > > >> >> > > > >>> > > > > > > > > waiting for the discussion/ticket and >>> > > proceed >>> > > >> >> with >>> > > >> >> > > the >>> > > >> >> > > > >>> > > > > contribution). >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > Kind regards, >>> > > >> >> > > > >>> > > > > > > > > Emmanouil >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > [1] >>> > > >> >> https://cloud.google.com/pubsub/docs/overview >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > *Emmanouil Gkatziouras* >>> > > >> >> > > > >>> > > > > > > > > https://egkatzioura.com/ | >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > >> >> > > > >>> > > > > > > > > https://github.com/gkatzioura >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis >>> > Magda < >>> > > >> >> > > > >>> dma...@apache.org >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > > > > wrote: >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > Hello Emmanouil, >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > Are you referring to any specific >>> > > >> >> implementation >>> > > >> >> > > > >>> tailored >>> > > >> >> > > > >>> > > for a >>> > > >> >> > > > >>> > > > > > > cloud? >>> > > >> >> > > > >>> > > > > > > > If >>> > > >> >> > > > >>> > > > > > > > > > I'm not mistaken you were >>> mentioning >>> > > >> AWS/GCE. >>> > > >> >> Is >>> > > >> >> > > > there >>> > > >> >> > > > >>> any >>> > > >> >> > > > >>> > > > Github >>> > > >> >> > > > >>> > > > > > > > source >>> > > >> >> > > > >>> > > > > > > > > > code? >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > Here is a list of existing >>> integrations >>> > > >> with >>> > > >> >> > > > streaming >>> > > >> >> > > > >>> and >>> > > >> >> > > > >>> > > > > pub/sub >>> > > >> >> > > > >>> > > > > > > > > > frameworks: >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> https://apacheignite-mix.readme.io/docs/overview >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > with the source code in Ignite >>> master: >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > https://github.com/apache/ignite/tree/master/modules >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > Check out those resources to see >>> what >>> > are >>> > > >> the >>> > > >> >> > most >>> > > >> >> > > > >>> suitable >>> > > >> >> > > > >>> > > > > Ignite >>> > > >> >> > > > >>> > > > > > > APIs >>> > > >> >> > > > >>> > > > > > > > > for >>> > > >> >> > > > >>> > > > > > > > > > such integrations. >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > @Saikat Maitra < >>> > saikat.mai...@gmail.com> >>> > > , >>> > > >> >> > @Roman >>> > > >> >> > > < >>> > > >> >> > > > >>> > > > > > rsht...@yahoo.com >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > > , >>> > > >> >> > > > >>> > > > > > > > > as >>> > > >> >> > > > >>> > > > > > > > > > streaming technologies >>> > > >> >> contributors/maintainers, >>> > > >> >> > > > could >>> > > >> >> > > > >>> you >>> > > >> >> > > > >>> > > > please >>> > > >> >> > > > >>> > > > > > > join >>> > > >> >> > > > >>> > > > > > > > > this >>> > > >> >> > > > >>> > > > > > > > > > thread? >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > - >>> > > >> >> > > > >>> > > > > > > > > > Denis >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM >>> > Emmanouil >>> > > >> >> > > > Gkatziouras < >>> > > >> >> > > > >>> > > > > > > > > > gkatzio...@gmail.com> >>> > > >> >> > > > >>> > > > > > > > > > wrote: >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > > Hello everyone, >>> > > >> >> > > > >>> > > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > > I started using Ignite lately. >>> Part >>> > of >>> > > my >>> > > >> >> work >>> > > >> >> > > > >>> involves >>> > > >> >> > > > >>> > > > > Pub/Sub, >>> > > >> >> > > > >>> > > > > > > > thus I >>> > > >> >> > > > >>> > > > > > > > > > > created a pub/sub streamer. >>> > > >> >> > > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka. >>> > > >> >> > > > >>> > > > > > > > > > > Here's how the streamer works. >>> Every >>> > > node >>> > > >> >> > creates >>> > > >> >> > > > an >>> > > >> >> > > > >>> > > instance >>> > > >> >> > > > >>> > > > > of >>> > > >> >> > > > >>> > > > > > a >>> > > >> >> > > > >>> > > > > > > > > > Pub/Sub >>> > > >> >> > > > >>> > > > > > > > > > > client. >>> > > >> >> > > > >>> > > > > > > > > > > The clients subscribe to the same >>> > topic >>> > > >> >> using >>> > > >> >> > the >>> > > >> >> > > > >>> same >>> > > >> >> > > > >>> > > > > > subscription >>> > > >> >> > > > >>> > > > > > > > > name. >>> > > >> >> > > > >>> > > > > > > > > > > A message is being sent and one >>> of >>> > the >>> > > >> nodes >>> > > >> >> > will >>> > > >> >> > > > >>> receive >>> > > >> >> > > > >>> > > the >>> > > >> >> > > > >>> > > > > > > > message, >>> > > >> >> > > > >>> > > > > > > > > > > process it and then forward it to >>> > > Ignite >>> > > >> >> using >>> > > >> >> > > the >>> > > >> >> > > > >>> > > extractor >>> > > >> >> > > > >>> > > > > > > > provided. >>> > > >> >> > > > >>> > > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > > Any thoughts? >>> > > >> >> > > > >>> > > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras* >>> > > >> >> > > > >>> > > > > > > > > > > https://egkatzioura.com/ | >>> > > >> >> > > > >>> > > > > > > > > > > >>> > > >> >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ >>> > > >> >> > > > >>> > > > > > > > > > > https://github.com/gkatzioura >>> > > >> >> > > > >>> > > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > > >>> > > >> >> > > > >>> > > > > > > > >>> > > >> >> > > > >>> > > > > > > >>> > > >> >> > > > >>> > > > > > >>> > > >> >> > > > >>> > > > > >>> > > >> >> > > > >>> > > > >>> > > >> >> > > > >>> > > >>> > > >> >> > > > >>> > >>> > > >> >> > > > >>> >>> > > >> >> > > > >> >>> > > >> >> > > > >>> > > >> >> > > >>> > > >> >> > >>> > > >> >> >>> > > >> > >>> > > >> >>> > > > >>> > > >>> > >>> >>
