Taking a Break From Community
Hi All, I will be taking a break from the Ignite community for an indefinite period. During this period, I will not be available on the dev list or looking at JIRA tickets. If anything urgent comes up, please email me on my Apache email ID, or reach out on Slack. Atri
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
A cloud deployment is not needed -- you only need to set the IPFinder to Azure and deploy a local Ignite cluster. If it boots up correctly, it works fine. I did not respond because my time is severely limited these days -- please note that my employment does not incorporate contributions to Ignite. Atri On Tue, Sep 7, 2021 at 5:28 PM Pavel Pereslegin wrote: > > Denis, > > if Atri looked at the changes why didn't he left any comments or > review approval? > > Only the ipfinder test was checked in Azure, it is essentially > synthetic, is it enough? As far as I understand, we have to deploy > Ignite itself in the cloud and play with it to make sure it works, > unfortunately, I do not have the competence, as well as free time to > study how to do this, so we asked for help. > > вт, 7 сент. 2021 г. в 14:23, Denis Magda : > > > > Maxim, > > > > Regardless of a lack of automated testing in a real environment, our AWS, > > GCE and other modules are used actively by the users and we don't receive > > issue reports. Here are the numbers: > > > >- AWS - 7000 *monthly* Maven downloads on average > >- GCE - 350 downloads > >- Kubernetes - 6000 downloads > > > > So they are far from dead. > > > > I think it's not corrected to skip the Azure integration from the release > > only because it follows the same testing approach. If you propose to change > > it, then let's do it outside of this discussion and for all the modules. > > The first step would be to find a sponsor who is ready to provision > > resources on the cloud environments (that was the main obstacle when the > > current test suites were being created). > > > > I don't know, currently, I'm not able to verify the results of this > > > execution. > > > > > > The module was verified by Atri and Pavel. This looks enough. If you have > > doubts, you can also double-check the code and work with Atri in case any > > issue arises. > > > > -- > > Denis > > > > - > > Denis > > > > On Mon, Sep 6, 2021 at 2:50 PM Maxim Muzafarov wrote: > > > > > Atri, > > > > > > I don't know, currently, I'm not able to verify the results of this > > > execution. For instance, we have the TcpDiscoveryZookeeperIpFinder and > > > it has the corresponding test suite on the TeamCity server, so when I > > > make code changes I can verify these changes on the test environment. > > > How can we be sure that new changes in TCP discovery Ignite's > > > component won't break the Azure module? I don't think we should allow > > > such a practice at all, so the AWS and GCE is not a good example here. > > > Probably we don't know are they work correctly at all because they are > > > don't even used by users. > > > > > > On Mon, 6 Sept 2021 at 21:18, Atri Sharma wrote: > > > > > > > > Maxim, > > > > > > > > I thought Pavel has already reported that the tests run fine with a > > > > valid Azure account? > > > > > > > > On Mon, Sep 6, 2021 at 11:10 PM Maxim Muzafarov > > > wrote: > > > > > > > > > > Folks, > > > > > > > > > > Sorry for the reminder :-) > > > > > > > > > > This issue [1] still blocks the release and the development. There is > > > > > a week that has passed, however, the issue is still not resolved and > > > > > cannot be validated within a reasonable time frame. Trust is a good > > > > > thing and all the community members should trust each other, but we > > > > > still have the RTC procedure and I don't think we should refuse оf it > > > > > just because of trust and all of us are highly qualified engineers. > > > > > This is my humble opinion, but I don't think that this issue [1] is a > > > > > good candidate to include the release since additional checks are > > > > > still required. Things like this can quickly turn into dead code (like > > > > > the AWS and GCE modules do) and it's better to add additional tests > > > > > for it prior to release them. > > > > > > > > > > So here is the plan: > > > > > > > > > > - revert [1] from the release 2.11 branch (I'll do it tomorrow) > > > > > - commit the [1] to the master branch (success build is enough > > > > > verification for this at this moment) > > > > > - create and configure TC with a te
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
Maxim, I thought Pavel has already reported that the tests run fine with a valid Azure account? On Mon, Sep 6, 2021 at 11:10 PM Maxim Muzafarov wrote: > > Folks, > > Sorry for the reminder :-) > > This issue [1] still blocks the release and the development. There is > a week that has passed, however, the issue is still not resolved and > cannot be validated within a reasonable time frame. Trust is a good > thing and all the community members should trust each other, but we > still have the RTC procedure and I don't think we should refuse оf it > just because of trust and all of us are highly qualified engineers. > This is my humble opinion, but I don't think that this issue [1] is a > good candidate to include the release since additional checks are > still required. Things like this can quickly turn into dead code (like > the AWS and GCE modules do) and it's better to add additional tests > for it prior to release them. > > So here is the plan: > > - revert [1] from the release 2.11 branch (I'll do it tomorrow) > - commit the [1] to the master branch (success build is enough > verification for this at this moment) > - create and configure TC with a technical azure account to test such module > - initiate a discussion to move Azure, AWS, GCE modules to the > ignite-extension project > - do everything above prior to the 2.12 release. > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > > > On Fri, 3 Sept 2021 at 18:36, Pavel Pereslegin wrote: > > > > Atri, > > > > > Anyone with azure account should be able to test it > > > > I checked the test included in the module with an Azure account (as I > > mentioned in Jira), it passes successfully. But I'm not sure if this > > is enough. > > > > пт, 3 сент. 2021 г. в 17:48, Nikolay Izhikov : > > > > > > Atri. > > > > > > Right now code can’t be tested because it doesn’t compile. > > > > > > > 3 сент. 2021 г., в 17:42, Atri Sharma написал(а): > > > > > > > > I am not sure why is it can be tested by few members? > > > > > > > > Anyone with azure account should be able to test it > > > > > > > > On Fri, 3 Sep 2021, 19:29 Maxim Muzafarov, wrote: > > > > > > > >> Folks, > > > >> > > > >> The GCP and AWS should also be moved to extensions, but this is a > > > >> discussion for another topic. > > > >> > > > >> I trust you all :-) > > > >> But who can validate the patch [1]? This is a bad practice that only a > > > >> few members be able to test the code. > > > >> > > > >> > > > >> [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > > >> > > > >> On Fri, 3 Sept 2021 at 16:43, Atri Sharma wrote: > > > >>> > > > >>> It was extensively tested by myself (running the tests on an actual > > > >>> Azure > > > >>> account and running an Ignite cluster using an Azure account) and Ilya > > > >>> (thanks Ilya!) prior to merging it in the core > > > >>> > > > >>> On Fri, 3 Sep 2021, 18:50 Denis Magda, wrote: > > > >>> > > > >>>> Disagree to exclude this contribution from the 2.11 release. As Atri > > > >>>> explained, its implementation and testing approach is identical to > > > >>>> the > > > >> AWS, > > > >>>> GCP and Kubernetes IP finders. > > > >>>> If we want to move the modules to extensions and improve the testing > > > >>>> approach, then it needs to be done for all similar components. > > > >>>> > > > >>>> Atri, have you tested the module in the cloud? If it works, then I > > > >> don't > > > >>>> see any reason why we need to have someone else double-checking this > > > >> once > > > >>>> again. This is the community, we need to trust each other. > > > >>>> > > > >>>> - > > > >>>> Denis > > > >>>> > > > >>>> On Fri, Sep 3, 2021 at 9:15 AM Atri Sharma wrote: > > > >>>> > > > >>>>> I would argue that the module Co exists with the other IP discovery > > > >>>> modules > > > >>>>> (such as GCP and AWS), which are part of core. > > > >>>>> > > > >&
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
I am not sure why is it can be tested by few members? Anyone with azure account should be able to test it On Fri, 3 Sep 2021, 19:29 Maxim Muzafarov, wrote: > Folks, > > The GCP and AWS should also be moved to extensions, but this is a > discussion for another topic. > > I trust you all :-) > But who can validate the patch [1]? This is a bad practice that only a > few members be able to test the code. > > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > On Fri, 3 Sept 2021 at 16:43, Atri Sharma wrote: > > > > It was extensively tested by myself (running the tests on an actual Azure > > account and running an Ignite cluster using an Azure account) and Ilya > > (thanks Ilya!) prior to merging it in the core > > > > On Fri, 3 Sep 2021, 18:50 Denis Magda, wrote: > > > > > Disagree to exclude this contribution from the 2.11 release. As Atri > > > explained, its implementation and testing approach is identical to the > AWS, > > > GCP and Kubernetes IP finders. > > > If we want to move the modules to extensions and improve the testing > > > approach, then it needs to be done for all similar components. > > > > > > Atri, have you tested the module in the cloud? If it works, then I > don't > > > see any reason why we need to have someone else double-checking this > once > > > again. This is the community, we need to trust each other. > > > > > > - > > > Denis > > > > > > On Fri, Sep 3, 2021 at 9:15 AM Atri Sharma wrote: > > > > > > > I would argue that the module Co exists with the other IP discovery > > > modules > > > > (such as GCP and AWS), which are part of core. > > > > > > > > In terms of tests, the azure module has exactly the same type of > tests as > > > > the other two modules mentioned above. > > > > > > > > On Fri, 3 Sep 2021, 17:54 Maxim Muzafarov, > wrote: > > > > > > > > > Folks, > > > > > > > > > > The patch [1] for the azure module seems to be ready for review, > > > > > however, there are still some questions that exist: > > > > > - Do we have a technical account in azure to test the moude on TC? > > > > > Manual testing for such changes is really annoying. > > > > > - Is there any reason to add these changes to the main Apache > Ignite > > > > > project? Why not extensions? > > > > > - It seems there is a lack of tests for this module. > > > > > > > > > > According to the mentions above, what do you think about moving > this > > > > > functionality to the next 2.12 release to solve all the questions > > > > > without a rush? I propose to revert the changes related to the > azure > > > > > functionality from the 2.11 branch. > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > > > > > > > > > On Fri, 27 Aug 2021 at 20:53, Dmitriy Pavlov > > > wrote: > > > > > > > > > > > > We had a separate discussion with Max and Alex, so I tend to > agree > > > that > > > > > > both issues are blockers. > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > пт, 27 авг. 2021 г. в 16:41, Alexey Gidaspov < > olive.c...@gmail.com>: > > > > > > > > > > > > > Hi, Maxim! > > > > > > > > > > > > > > These issues really block release. So we should wait fixes. > > > > > > > > > > > > > > On 2021/08/27 12:46:24, Maxim Muzafarov > wrote: > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > > > > > I've been faced these issues, which seems to be a blocker > for the > > > > > > > > release. Please, take a look and share your thoughts. > > > > > > > > > > > > > > > > The server node fails when the client node with the higher > java > > > > > > > > version connects to it [1]. > > > > > > > > The Apache Ignite build fails with missing dependency [2]. > > > > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jir
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
It was extensively tested by myself (running the tests on an actual Azure account and running an Ignite cluster using an Azure account) and Ilya (thanks Ilya!) prior to merging it in the core On Fri, 3 Sep 2021, 18:50 Denis Magda, wrote: > Disagree to exclude this contribution from the 2.11 release. As Atri > explained, its implementation and testing approach is identical to the AWS, > GCP and Kubernetes IP finders. > If we want to move the modules to extensions and improve the testing > approach, then it needs to be done for all similar components. > > Atri, have you tested the module in the cloud? If it works, then I don't > see any reason why we need to have someone else double-checking this once > again. This is the community, we need to trust each other. > > - > Denis > > On Fri, Sep 3, 2021 at 9:15 AM Atri Sharma wrote: > > > I would argue that the module Co exists with the other IP discovery > modules > > (such as GCP and AWS), which are part of core. > > > > In terms of tests, the azure module has exactly the same type of tests as > > the other two modules mentioned above. > > > > On Fri, 3 Sep 2021, 17:54 Maxim Muzafarov, wrote: > > > > > Folks, > > > > > > The patch [1] for the azure module seems to be ready for review, > > > however, there are still some questions that exist: > > > - Do we have a technical account in azure to test the moude on TC? > > > Manual testing for such changes is really annoying. > > > - Is there any reason to add these changes to the main Apache Ignite > > > project? Why not extensions? > > > - It seems there is a lack of tests for this module. > > > > > > According to the mentions above, what do you think about moving this > > > functionality to the next 2.12 release to solve all the questions > > > without a rush? I propose to revert the changes related to the azure > > > functionality from the 2.11 branch. > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > > > > > On Fri, 27 Aug 2021 at 20:53, Dmitriy Pavlov > wrote: > > > > > > > > We had a separate discussion with Max and Alex, so I tend to agree > that > > > > both issues are blockers. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пт, 27 авг. 2021 г. в 16:41, Alexey Gidaspov : > > > > > > > > > Hi, Maxim! > > > > > > > > > > These issues really block release. So we should wait fixes. > > > > > > > > > > On 2021/08/27 12:46:24, Maxim Muzafarov wrote: > > > > > > Folks, > > > > > > > > > > > > > > > > > > I've been faced these issues, which seems to be a blocker for the > > > > > > release. Please, take a look and share your thoughts. > > > > > > > > > > > > The server node fails when the client node with the higher java > > > > > > version connects to it [1]. > > > > > > The Apache Ignite build fails with missing dependency [2]. > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14725 > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-15388 > > > > > > > > > > > > On Mon, 9 Aug 2021 at 16:56, Maxim Muzafarov > > > wrote: > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > On Mon, 9 Aug 2021 at 16:27, Ivan Daschinsky < > > ivanda...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > I suppose, that > > > https://issues.apache.org/jira/browse/IGNITE-15274 > > > > > is a > > > > > > > > blocker. > > > > > > > > Fix is quite easy and straightforward > > > > > > > > > > > > > > > > пн, 2 авг. 2021 г. в 03:11, Igor Sapego >: > > > > > > > > > > > > > > > > > Cherry-picked to release branch. > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 30, 2021 at 3:11 PM Alexey Gidaspov <
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
I would argue that the module Co exists with the other IP discovery modules (such as GCP and AWS), which are part of core. In terms of tests, the azure module has exactly the same type of tests as the other two modules mentioned above. On Fri, 3 Sep 2021, 17:54 Maxim Muzafarov, wrote: > Folks, > > The patch [1] for the azure module seems to be ready for review, > however, there are still some questions that exist: > - Do we have a technical account in azure to test the moude on TC? > Manual testing for such changes is really annoying. > - Is there any reason to add these changes to the main Apache Ignite > project? Why not extensions? > - It seems there is a lack of tests for this module. > > According to the mentions above, what do you think about moving this > functionality to the next 2.12 release to solve all the questions > without a rush? I propose to revert the changes related to the azure > functionality from the 2.11 branch. > > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > On Fri, 27 Aug 2021 at 20:53, Dmitriy Pavlov wrote: > > > > We had a separate discussion with Max and Alex, so I tend to agree that > > both issues are blockers. > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 27 авг. 2021 г. в 16:41, Alexey Gidaspov : > > > > > Hi, Maxim! > > > > > > These issues really block release. So we should wait fixes. > > > > > > On 2021/08/27 12:46:24, Maxim Muzafarov wrote: > > > > Folks, > > > > > > > > > > > > I've been faced these issues, which seems to be a blocker for the > > > > release. Please, take a look and share your thoughts. > > > > > > > > The server node fails when the client node with the higher java > > > > version connects to it [1]. > > > > The Apache Ignite build fails with missing dependency [2]. > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14725 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-15388 > > > > > > > > On Mon, 9 Aug 2021 at 16:56, Maxim Muzafarov > wrote: > > > > > > > > > > +1 > > > > > > > > > > On Mon, 9 Aug 2021 at 16:27, Ivan Daschinsky > > > wrote: > > > > > > > > > > > > I suppose, that > https://issues.apache.org/jira/browse/IGNITE-15274 > > > is a > > > > > > blocker. > > > > > > Fix is quite easy and straightforward > > > > > > > > > > > > пн, 2 авг. 2021 г. в 03:11, Igor Sapego : > > > > > > > > > > > > > Cherry-picked to release branch. > > > > > > > > > > > > > > Best Regards, > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 30, 2021 at 3:11 PM Alexey Gidaspov < > > > olive.c...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Ok, I think we should add [1] to 2.11 release. Can you > > > cherry-pick it to > > > > > > > > release branch? > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14815 > > > > > > > > > > > > > > > > On 2021/07/30 02:25:25, Igor Sapego > wrote: > > > > > > > > > I'm fine with any of these two solutions. > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 6:36 PM Ilya Kasnacheev < > > > > > > > > ilya.kasnach...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > > > > > > > I think it does make sense to include changes which will > let > > > us avoid > > > > > > > > > > migration issues in the future. > > > > > > > > > > > > > > > > > > > > Alternatively, maybe we can revert the incomplete change > > > from 2.11 in > > > > > > > > order > > > > > > > > > > to reintroduce it in 2.12 in full form if Igor agrees > and RE > > > thinks > > > > > > > > this is > > > > > > > > > > the best course of action. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > -- > > > > > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > чт, 29 июл. 2021 г. в 18:07, Igor Sapego < > isap...@apache.org > > > >: > > > > > > > > > > > > > > > > > > > > > Alexey, > > > > > > > > > > > > > > > > > > > > > > It contains changes we could not introduce if we > release > > > ignite in > > > > > > > > its > > > > > > > > > > > current state as they are going to be breaking changes. > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:13 PM Alexey Gidaspov < > > > > > > > > olive.c...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, Igor! > > > > > > > > > > > > > > > > > > > > > > > > Now we are in stabilization phase and accepting only > > > blockers. I > > > > > > > > may be > > > > > > > > > > > > wrong, but this ticket doesn't seem to be of that > kind. > > > > > > > > > > > > > > > > > > > > > > > > On 2021/07/28 21:00:15, Igor Sapego < > isap...@apache.org> > > > wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > > > > > I suggest adding [1] to
Re: Build failed on the ignite-azure module. Missing dependency in maven central repository
I will review tonight. On Wed, 1 Sep 2021, 19:54 Pavel Pereslegin, wrote: > Hello, Igniters! > I think the patch is ready [1]. > > Ilya, Atri, > could you help with the review? > > [1] https://issues.apache.org/jira/browse/IGNITE-15388 > > пт, 27 авг. 2021 г. в 15:29, Maxim Muzafarov : > > > > Folks, > > > > I've created an issue to fix this. I think this is a blocker for the > > 2.11 release since there is no workaround here. > > https://issues.apache.org/jira/browse/IGNITE-15388 > > > > > > On Fri, 27 Aug 2021 at 11:59, Maxim Muzafarov wrote: > > > > > > I've found the recommendation to bump up the version SDK [1]. > > > > > > [1] > https://github.com/Azure/azure-sdk-for-java/issues/23562#issuecomment-903183704 > > > > > > On Fri, 27 Aug 2021 at 11:27, Petr Ivanov wrote: > > > > > > > > Teamcity will be able to build release — we have caching proxy with > Artifactory which seems has cached that artifact already. > > > > I think I can provide it as a file to republish to mvnrepository, > but I have no such permissions. > > > > > > > > Who could help us with uploading it? > > > > > > > > > > > > > On 27 Aug 2021, at 11:23, Maxim Muzafarov > wrote: > > > > > > > > > > Folks, > > > > > > > > > > I've recently tried to build the Apache Ignite project from scratch > > > > > and got the following error: > > > > > > > > > > [INFO] > > > > > > [INFO] BUILD FAILURE > > > > > [INFO] > > > > > > [INFO] Total time: 01:27 min > > > > > [INFO] Finished at: 2021-08-25T22:40:58+03:00 > > > > > [INFO] > > > > > > [ERROR] Failed to execute goal on project ignite-azure: Could not > > > > > resolve dependencies for project > > > > > org.apache.ignite:ignite-azure:jar:2.12.0-SNAPSHOT: Failure to find > > > > > com.azure:azure-storage-blob:jar:12.0.0 in > > > > > https://repo.maven.apache.org/maven2 was cached in the local > > > > > repository, resolution will not be reattempted until the update > > > > > interval of central has elapsed or updates are forced -> [Help 1] > > > > > > > > > > It seems the azure-storage-blob is missing in the maven central > > > > > repository [1]. The mirror is not available since the bintray was > > > > > deprecated in May [2]. > > > > > > > > > > What should we do with this? I see the following: > > > > > - bump up the version of the azure module (compilation error > occurs, > > > > > some fix required) > > > > > - republish the 12.0.0 version to the maven. It seems it should be > available [3] > > > > > > > > > > How it affects the 2.11 release? Should we file an issue if we > need to > > > > > bump up the version? > > > > > > > > > > [1] > https://repo1.maven.org/maven2/com/azure/azure-storage-blob/12.0.0 > > > > > [2] https://jcenter.bintray.com/ > > > > > [3] > https://mvnrepository.com/artifact/com.azure/azure-storage-blob/12.0.0 > > > > >
Re: [Announcement] Apache Ignite 2.11 Code Freeze started
I will look into 15388 On Fri, 27 Aug 2021, 18:16 Maxim Muzafarov, wrote: > Folks, > > > I've been faced these issues, which seems to be a blocker for the > release. Please, take a look and share your thoughts. > > The server node fails when the client node with the higher java > version connects to it [1]. > The Apache Ignite build fails with missing dependency [2]. > > > [1] https://issues.apache.org/jira/browse/IGNITE-14725 > [2] https://issues.apache.org/jira/browse/IGNITE-15388 > > On Mon, 9 Aug 2021 at 16:56, Maxim Muzafarov wrote: > > > > +1 > > > > On Mon, 9 Aug 2021 at 16:27, Ivan Daschinsky > wrote: > > > > > > I suppose, that https://issues.apache.org/jira/browse/IGNITE-15274 is > a > > > blocker. > > > Fix is quite easy and straightforward > > > > > > пн, 2 авг. 2021 г. в 03:11, Igor Sapego : > > > > > > > Cherry-picked to release branch. > > > > > > > > Best Regards, > > > > Igor > > > > > > > > > > > > On Fri, Jul 30, 2021 at 3:11 PM Alexey Gidaspov < > olive.c...@gmail.com> > > > > wrote: > > > > > > > > > Ok, I think we should add [1] to 2.11 release. Can you cherry-pick > it to > > > > > release branch? > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14815 > > > > > > > > > > On 2021/07/30 02:25:25, Igor Sapego wrote: > > > > > > I'm fine with any of these two solutions. > > > > > > > > > > > > Best Regards, > > > > > > Igor > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 6:36 PM Ilya Kasnacheev < > > > > > ilya.kasnach...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > I think it does make sense to include changes which will let > us avoid > > > > > > > migration issues in the future. > > > > > > > > > > > > > > Alternatively, maybe we can revert the incomplete change from > 2.11 in > > > > > order > > > > > > > to reintroduce it in 2.12 in full form if Igor agrees and RE > thinks > > > > > this is > > > > > > > the best course of action. > > > > > > > > > > > > > > Regards, > > > > > > > -- > > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > > > > чт, 29 июл. 2021 г. в 18:07, Igor Sapego : > > > > > > > > > > > > > > > Alexey, > > > > > > > > > > > > > > > > It contains changes we could not introduce if we release > ignite in > > > > > its > > > > > > > > current state as they are going to be breaking changes. > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 29, 2021 at 4:13 PM Alexey Gidaspov < > > > > > olive.c...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, Igor! > > > > > > > > > > > > > > > > > > Now we are in stabilization phase and accepting only > blockers. I > > > > > may be > > > > > > > > > wrong, but this ticket doesn't seem to be of that kind. > > > > > > > > > > > > > > > > > > On 2021/07/28 21:00:15, Igor Sapego > wrote: > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > I suggest adding [1] to the scope of release, because it > > > > contains > > > > > > > > > > changes to code introduced by [2], which is already > included in > > > > > > > > release. > > > > > > > > > > > > > > > > > > > > [1] - https://issues.apache.org/jira/browse/IGNITE-14815 > > > > > > > > > > [2] - https://issues.apache.org/jira/browse/IGNITE-14658 > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > Igor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 11:30 PM Maxim Muzafarov < > > > > > mmu...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > > > The issues mentioned above were successfully resolved > and > > > > > > > > > > > cherry-picked to the ignite-2.11 branch. Sorry for the > > > > delay. I > > > > > > > think > > > > > > > > > > > we can proceed with the release. > > > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jul 2021 at 15:44, Maxim Muzafarov < > > > > > mmu...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > > > > > > > Yes, both the fixes [1] [2] will not require > performance > > > > > tests > > > > > > > > rerun. > > > > > > > > > > > > > > > > > > > > > > > > [1] > https://issues.apache.org/jira/browse/IGNITE-15146 > > > > > > > > > > > > [2] > https://issues.apache.org/jira/browse/IGNITE-15170 > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jul 2021 at 15:30, Dmitry Pavlov < > > > > > dpav...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > I personally trust opinion of Nikolay and Maxim, > we can > > > > > > > consider > > > > > > > > > both > > > > > > > > > > > as blockers. > > > > > > > > > > > > > > > > > > > > > > > > > > Just an idea to consider: > > > > > > > > > > > > > For fixed ticket/PR ( > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-15170) > most > > > > > likely > > > > > >
Re: Text Queries Support
Hi Ivan, I didn't quite understand. Are you proposing that Ignite should not have FTS capabilities? Atri On Tue, Aug 3, 2021 at 2:57 PM Ivan Pavlukhin wrote: > > Hi Atri, > > My main concern is non-maleficence. Every task has several solutions, > e.g. straightforward ones: > 1. Do not implement FTS. > 2. Create own implementation. > > Some of the strongest ones live without FTS [1]. > > [1] https://github.com/cockroachdb/cockroach/issues/7821 > > 2021-08-02 11:33 GMT+03:00, Atri Sharma : > > Hi Ivan, > > > > Would you like to propose an alternative to Lucene? > > > > Atri > > > > On Mon, 2 Aug 2021, 13:48 Ivan Pavlukhin, wrote: > > > >> Folks, > >> > >> Sorry if read the thread not thoroughly enough, but do we consider > >> Lucene as obviously right choice? In my understanding Ignite history > >> has shown clearly that "fastest feature implementation" is not usually > >> the best. And one example of this are text queries. Are not we trying > >> to do a same mistake again? FTS is a huge feature, I do not believe > >> there is an easy win for it. > >> > >> 2021-07-27 19:18 GMT+03:00, Atri Sharma : > >> > Andrey, > >> > > >> >> Per-partition Lucene index looks simple to implement, but it may > >> >> require > >> >> per-partition SQL to make full-text search expressions work correctly > >> >> within the SQL quiery. > >> > I think that as long as we follow the map - reduce process that we > >> > already do for other queries, we should be fine. > >> > > >> >> Per-partition SQL index may kill the performance. We already tried to > >> >> do > >> >> that in Ignite 2. However, QueryParallelism feature helps to speed up > >> >> some > >> >> data-intensive queries, > >> >> but hits the performance in simple cases, and at some point (e.g. > >> >> segments > >> >> > number of CPU) the performance rapidly degrades with the increasing > >> >> number of segments. > >> > > >> > Yeah, that is always the case, but a global index will be a nightmare > >> > in terms of concurrency and pessimistic concurrency control will > >> > anyways kill the benefits, coupled with the metadata requirements. > >> > What were the specific issues with per partition index? > >> >> > >> >> AFAIK, Lucene widely used bitmap indices that are easy to merge. > >> >> Maybe, the map-reduce technique underneath FTS expressions and some > >> hacks > >> >> will add a minimal overhead. > >> > > >> > Lucene uses many types of indices but the aspect here is that per > >> > partition Lucene indices can return docIDs and we can merge them in > >> > reduce phase. So we are abstracted out from specifics of the internal > >> > index being used to serve the query. > >> > > >> >> > >> >> > As illustrated by Ilya, we can use Ignite's WAL records to rebuild > >> >> > Lucene indices. The important thing here is to not treat Lucene > >> >> > indices as source of truth. > >> >> To use WAL we either should relay Lucene files to our Page memory or > >> >> be > >> >> aware of Lucene files structure. > >> >> The first looks tricky, as we should guarantee a contiguous address > >> space > >> >> in Page memory for reflecting Lucene file. Maybe separate managed > >> >> memory > >> >> segment with its own rules? > >> > > >> > Why not use Lucene's MMappedDirectory and map it to our storage > >> > classes? > >> > > >> >> > >> >> >> Transactions. > >> >> >> * Will we support transactions? > >> >> > Lucene has no concept of transactions. > >> >> Yes, but we have. > >> >> Lucene index may be non-transactional, but users never expect to see > >> >> uncommited data. > >> >> How does this connect with transactional SQL? > >> > We could have the Lucene writes done as a part of transactions and ack > >> > back only when it succeeds/fails. WDYT? > >> >> > >> >> On Tue, Jul 27, 2021 at 1:36 PM Atri Sharma wrote: > >> >> > >> >> > Sorry, I planned on creating a Wiki page for this, but it makes more &g
Re: Text Queries Support
Hi Ivan, Would you like to propose an alternative to Lucene? Atri On Mon, 2 Aug 2021, 13:48 Ivan Pavlukhin, wrote: > Folks, > > Sorry if read the thread not thoroughly enough, but do we consider > Lucene as obviously right choice? In my understanding Ignite history > has shown clearly that "fastest feature implementation" is not usually > the best. And one example of this are text queries. Are not we trying > to do a same mistake again? FTS is a huge feature, I do not believe > there is an easy win for it. > > 2021-07-27 19:18 GMT+03:00, Atri Sharma : > > Andrey, > > > >> Per-partition Lucene index looks simple to implement, but it may require > >> per-partition SQL to make full-text search expressions work correctly > >> within the SQL quiery. > > I think that as long as we follow the map - reduce process that we > > already do for other queries, we should be fine. > > > >> Per-partition SQL index may kill the performance. We already tried to do > >> that in Ignite 2. However, QueryParallelism feature helps to speed up > >> some > >> data-intensive queries, > >> but hits the performance in simple cases, and at some point (e.g. > >> segments > >> > number of CPU) the performance rapidly degrades with the increasing > >> number of segments. > > > > Yeah, that is always the case, but a global index will be a nightmare > > in terms of concurrency and pessimistic concurrency control will > > anyways kill the benefits, coupled with the metadata requirements. > > What were the specific issues with per partition index? > >> > >> AFAIK, Lucene widely used bitmap indices that are easy to merge. > >> Maybe, the map-reduce technique underneath FTS expressions and some > hacks > >> will add a minimal overhead. > > > > Lucene uses many types of indices but the aspect here is that per > > partition Lucene indices can return docIDs and we can merge them in > > reduce phase. So we are abstracted out from specifics of the internal > > index being used to serve the query. > > > >> > >> > As illustrated by Ilya, we can use Ignite's WAL records to rebuild > >> > Lucene indices. The important thing here is to not treat Lucene > >> > indices as source of truth. > >> To use WAL we either should relay Lucene files to our Page memory or be > >> aware of Lucene files structure. > >> The first looks tricky, as we should guarantee a contiguous address > space > >> in Page memory for reflecting Lucene file. Maybe separate managed memory > >> segment with its own rules? > > > > Why not use Lucene's MMappedDirectory and map it to our storage classes? > > > >> > >> >> Transactions. > >> >> * Will we support transactions? > >> > Lucene has no concept of transactions. > >> Yes, but we have. > >> Lucene index may be non-transactional, but users never expect to see > >> uncommited data. > >> How does this connect with transactional SQL? > > We could have the Lucene writes done as a part of transactions and ack > > back only when it succeeds/fails. WDYT? > >> > >> On Tue, Jul 27, 2021 at 1:36 PM Atri Sharma wrote: > >> > >> > Sorry, I planned on creating a Wiki page for this, but it makes more > >> > sense to be replying here. > >> > > >> > > * How Lucene index can be split among the nodes? > >> > > >> > We can have partition level indices on each node. > >> > > >> > > * If we'll have a single index for all partitions on the particular > >> > > node, > >> > > then how index records will be aware of partitioning? > >> > > >> > Index records dont need to be aware of partitioning -- each Lucene > >> > index is independent. > >> > > >> > > This is important to filter out backup records from the results to > >> > > avoid > >> > > duplicates. > >> > > >> > We can merge documents from different nodes and remove duplicates as > >> > long as docIDs are globally unique. > >> > > >> > > * How results from several nodes can be merged on the Reduce stage? > >> > > >> > As long as documents have a globally unique docID, Lucene has merge > >> > functions that can merge results from multiple partial results. > >> > > >> > > * Does Lucene supports smth like JOIN operation or others that may > >>
Re: Apache Ignite 3 Alpha 2 webinar follow up questions
Query caching works on three levels - caching results, caching blocks and caching query plans. Prepared queries work by caching a plan for a query and reusing that plan by changing the parameters for the incoming query. So the query remains the same, but input values keep changing. The problem with prepared queries is that query execution can go bad very fast if the underlying data distribution changes and the cached plan is no longer optimal for the given statistics. On Sat, 31 Jul 2021, 12:54 Ivan Pavlukhin, wrote: > Hi Courtney, > > Please clarify what do you mean by prepared queries and query caching? > Do you mean caching query results? If so, in my mind material views > are the best approach here (Ignite 2 does not support them). Do you > have other good approaches in your mind? E.g. implemented in other > databases. > > 2021-07-26 21:27 GMT+03:00, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > Hi Courtney, > > > > Generally speaking, query caching certainly makes sense. As far as I > know, > > Ignite 2.x actually does that, but most likely there might be room for > > improvement as well. We will look into this. > > > > As for the SQL API - the answer is yes. The requirement for a dummy cache > > is an artifact of the current architecture. This is 100% wrong and will > be > > changed in 3.0. > > > > -Val > > > > On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson > > > > wrote: > > > >> Something else came to mind, are there plans to support prepared > queries? > >> > >> I recall someone saying before that Ignite does internally cache queries > >> but it's not at all clear if or how it does do that. I assume a simple > >> hash > >> of the query isn't enough. > >> > >> We generate SQL queries based on user runtime settings and they can get > >> to > >> hundreds of lines long, I imagine this means most of our queries are not > >> being cached but there are patterns so we could generate and manage > >> prepared queries ourselves. > >> > >> Also, will there be a dedicated API for doing SQL queries rather than > >> having to pass a SqlFieldsQuery to a cache that has nothing to do with > >> the > >> cache being queried? When I first started with Ignite years ago, this > was > >> beyond confusing for me. I'm trying to run select x from B but I pass > >> this > >> to a cache called DUMMY or whatever arbitrary name... > >> > >> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson < > >> courtney.robin...@hypi.io> > >> wrote: > >> > >> > Andrey, > >> > Thanks for the response - see my comments inline. > >> > > >> > > >> >> I've gone through the questions and have no the whole picture of your > >> use > >> >> case. > >> > > >> > Would you please clarify how you exactly use the Ignite? what are the > >> >> integration points? > >> >> > >> > > >> > I'll try to clarify - we have a low/no code platform. A user designs a > >> > model for their application and we map this model to Ignite tables and > >> > other data sources. The model I'll describe is what we're building now > >> and > >> > expected to be in alpha some time in Q4 21. Our current production > >> > architecture is different and isn't as generic, it is heavily tied to > >> > Ignite and we've redesigned to get some flexibility where Ignite > >> > doesn't > >> > provide what we want. Things like window functions and other SQL-99 > >> limits. > >> > > >> > In the next gen version we're working on you can create a model for a > >> > Tweet(content, to) and we will create an Ignite table with content and > >> > to > >> > columns using the type the user selects. This is the simplest case. > >> > We are adding generic support for sources and sinks and using Calcite > >> > as > >> a > >> > data virtualisation layer. Ignite is one of the available > source/sinks. > >> > > >> > When a user creates a model for Tweet, we also allow them to specify > >> > how > >> > they want to index the data. We have a copy of the calcite > >> > Elasticsearch > >> > adapter modified for Solr. > >> > > >> > When a source is queried (Ignite or any other that we support), we > >> > generate SQL that Calcite executes. Calcite will push down the > >> > generated > >> > queries to Solr and Solr produces a list of IDs (in case of Ignite) > and > >> we > >> > do a multi-get from Ignite to produce the actual results. > >> > > >> > Obviously there's a lot more to this but that should give you a > general > >> > idea. > >> > > >> > and maybe share some experience with using Ignite SPIs? > >> >> > >> > Our evolution with Ignite started from the key value + compute APIs. > We > >> > used the SPIs then but have since moved to using only the Ignite SQL > >> > API > >> > (we gave up transactions for this). > >> > > >> > We originally used the indexing SPI to keep our own lucene index of > >> > data > >> > in a cache. We did not use the Ignite FTS as it is very limited > >> > compared > >> to > >> > what we allow customers to do. If I remember correctly, we were using > >> > an > >>
Re: Review Requested -- IGNITE-15077
Hi Ilya, Following up on this please. On Tue, 27 Jul 2021, 22:20 Atri Sharma, wrote: > Hi Ilya, > > > > Frankly speaking, I do not see the value of having an extra layer of > > indirection around *local* Quartz-based scheduler in Ignite. Can you > > elaborate? > > I didnt quite understand that. Are you referring to the > IgniteCombinedSchedulerProcessor? > > > > Our guidelines also recommend having issue description to document the > whys > > and hows, and not just issue title. > > Sure, I will update the issue with more details. > > -- > Regards, > > Atri > Apache Concerted >
Re: Review Requested -- IGNITE-15077
Hi Ilya, > Frankly speaking, I do not see the value of having an extra layer of > indirection around *local* Quartz-based scheduler in Ignite. Can you > elaborate? I didnt quite understand that. Are you referring to the IgniteCombinedSchedulerProcessor? > > Our guidelines also recommend having issue description to document the whys > and hows, and not just issue title. Sure, I will update the issue with more details. -- Regards, Atri Apache Concerted
Re: Text Queries Support
Andrey, > Per-partition Lucene index looks simple to implement, but it may require > per-partition SQL to make full-text search expressions work correctly > within the SQL quiery. I think that as long as we follow the map - reduce process that we already do for other queries, we should be fine. > Per-partition SQL index may kill the performance. We already tried to do > that in Ignite 2. However, QueryParallelism feature helps to speed up some > data-intensive queries, > but hits the performance in simple cases, and at some point (e.g. segments > > number of CPU) the performance rapidly degrades with the increasing > number of segments. Yeah, that is always the case, but a global index will be a nightmare in terms of concurrency and pessimistic concurrency control will anyways kill the benefits, coupled with the metadata requirements. What were the specific issues with per partition index? > > AFAIK, Lucene widely used bitmap indices that are easy to merge. > Maybe, the map-reduce technique underneath FTS expressions and some hacks > will add a minimal overhead. Lucene uses many types of indices but the aspect here is that per partition Lucene indices can return docIDs and we can merge them in reduce phase. So we are abstracted out from specifics of the internal index being used to serve the query. > > > As illustrated by Ilya, we can use Ignite's WAL records to rebuild > > Lucene indices. The important thing here is to not treat Lucene > > indices as source of truth. > To use WAL we either should relay Lucene files to our Page memory or be > aware of Lucene files structure. > The first looks tricky, as we should guarantee a contiguous address space > in Page memory for reflecting Lucene file. Maybe separate managed memory > segment with its own rules? Why not use Lucene's MMappedDirectory and map it to our storage classes? > > >> Transactions. > >> * Will we support transactions? > > Lucene has no concept of transactions. > Yes, but we have. > Lucene index may be non-transactional, but users never expect to see > uncommited data. > How does this connect with transactional SQL? We could have the Lucene writes done as a part of transactions and ack back only when it succeeds/fails. WDYT? > > On Tue, Jul 27, 2021 at 1:36 PM Atri Sharma wrote: > > > Sorry, I planned on creating a Wiki page for this, but it makes more > > sense to be replying here. > > > > > * How Lucene index can be split among the nodes? > > > > We can have partition level indices on each node. > > > > > * If we'll have a single index for all partitions on the particular node, > > > then how index records will be aware of partitioning? > > > > Index records dont need to be aware of partitioning -- each Lucene > > index is independent. > > > > > This is important to filter out backup records from the results to avoid > > > duplicates. > > > > We can merge documents from different nodes and remove duplicates as > > long as docIDs are globally unique. > > > > > * How results from several nodes can be merged on the Reduce stage? > > > > As long as documents have a globally unique docID, Lucene has merge > > functions that can merge results from multiple partial results. > > > > > * Does Lucene supports smth like JOIN operation or others that may > > require > > > data from another partition or index? > > > > As illustrated by Ilya, Block-Join works for us. > > > > > If so, then it likes to multistep query with merging results on > > > intermediate stages and requires detailed investigation and design. > > > It is ok if Ignite will have some limitations here, but we would like to > > > know about them at the early stage. > > > > > * How effectively map Lucene files to the page memory? Is it even > > possible? > > > > Lucene has PageDirectory implementations which allow storing Lucene > > indices on different kind of file structures. It has a > > MMappedFileDirectory that we could use? > > > > > Otherwise, how to deal with potential OOM on large queries and memory > > > capacity planning? > > > > We can use Lucene's MMapped directory. > > > > > > > > Persistence. > > > * How and what consistency guarantees could we have/expect? > > > > Lucene does not have WAL logs but is append only > > > > > Seems, we may not be able to write physical records for Lucene index to > > our > > > WAL. What can we do with this? > > > > As illustrated by Ilya, we can use Ignite's WAL records to rebuild > > Lucene indices.
Review Requested -- IGNITE-15077
Hello, I have raised a PR for the said JIRA: https://github.com/apache/ignite/pull/9277 TC is green with the PR: https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite/6105253 Please help in reviewing. Regards, Atri
Re: Text Queries Support
Sorry, I planned on creating a Wiki page for this, but it makes more sense to be replying here. > * How Lucene index can be split among the nodes? We can have partition level indices on each node. > * If we'll have a single index for all partitions on the particular node, > then how index records will be aware of partitioning? Index records dont need to be aware of partitioning -- each Lucene index is independent. > This is important to filter out backup records from the results to avoid > duplicates. We can merge documents from different nodes and remove duplicates as long as docIDs are globally unique. > * How results from several nodes can be merged on the Reduce stage? As long as documents have a globally unique docID, Lucene has merge functions that can merge results from multiple partial results. > * Does Lucene supports smth like JOIN operation or others that may require > data from another partition or index? As illustrated by Ilya, Block-Join works for us. > If so, then it likes to multistep query with merging results on > intermediate stages and requires detailed investigation and design. > It is ok if Ignite will have some limitations here, but we would like to > know about them at the early stage. > * How effectively map Lucene files to the page memory? Is it even possible? Lucene has PageDirectory implementations which allow storing Lucene indices on different kind of file structures. It has a MMappedFileDirectory that we could use? > Otherwise, how to deal with potential OOM on large queries and memory > capacity planning? We can use Lucene's MMapped directory. > > Persistence. > * How and what consistency guarantees could we have/expect? Lucene does not have WAL logs but is append only > Seems, we may not be able to write physical records for Lucene index to our > WAL. What can we do with this? As illustrated by Ilya, we can use Ignite's WAL records to rebuild Lucene indices. The important thing here is to not treat Lucene indices as source of truth. > > Transactions. > * Will we support transactions? Lucene has no concept of transactions. > * Should Lucene be aware of Transaction and track mvcc (or whatever) > versions for the records? No > * What will be consistency guarantees? We can acknowledge writes back only after Lucene index is updated. > > UX > * How to add FullText search queries syntax into Calcite? Postgres's FTS functions are a good reference. > * AFAIK, the Lucene index has many properties for tuning. How will the user > configure the index? Most of those properties can be cluster level and exposed as a new sub config for ignite. > * How and where to store the settings? What are cluster-wide and what a > local to the particular node? All can be cluster level. > * Will be all the settings immutable? Can be they changed on-fly? after > node/grid restart? They should be applied post restart. > * Any limitations on query syntax? It depends on how we model our queries for text search. > > SQL > * Will we support FullText search in SQL? We need custom functions for it. See Postgres's FTS functions. > * How to integrate Lucene index into Calcite? What is the cost model? There cannot be any cost model since there are no paths for a text query. If we see a text query, we have to use Lucene index or return an error. In this way, we need to model text search as a set of UDFs > Splitting rules? Traits? Please see my reply above. > > > With all of this, you can go with the IEP (or even some short summary) and > further POC and implementation. > That's a big deal, so let's discuss what could be done here. > > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma wrote: > > > I am actually happy to drive the feature for Ignite 3. FTS is very > > important for me and I think Ignite users will benefit from it > > greatly. > > > > If it makes sense to be focusing on Ignite 3 for this capability, I am > > eager to contribute there and lead the development. > > > > Please share your thoughts. > > > > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov > > wrote: > > > > > > Hi Atri, > > > > > > All the Jira tickets we have on the Full-text search (FTS) thing are > > > targeted to Ignite 2. > > > > > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3, > > yet. > > > By the way, we are getting requests for this thing from the user side, > > and > > > definitely, > > > FTS would be a valuable feature for Ignite. > > > > > > It will be great if the one wants to drive it, any help will be > > appreciated. > > > > > > > > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma wrote: > > > > > >
Re: Text Queries Support
+1 Lets expose custom functions in Ignite SQL which allows us to use the full capabilities that Lucene offers On Mon, 26 Jul 2021, 21:51 Andrey Mashenkov, wrote: > Val, > > > I believe this is something we can look into in the scope of Ignite 3. > > Andrey, does Calcite have any support for this? What's your view on this? > > As Atri already mentioned, SQL 92 standard declares "LIKE" operator for > pattern matching. > Calcite supports LIKE operator. > > I've found it is a RexNode (expression) and I doubt it supports indices. > Maybe, LIKE can use a sorted index for prefix matching or equality > conditions, but it is very far from what we are talking about. > > Full-text search term is much wider than just a pattern matching. > Lucene provides much more capabilities on that and has rich > syntax contrary to "LIKE" operator. > So, LIKE operator is the standard operator with the defined contract. I'm > not sure it is worth integrating Lucene just for it. > I think we should have native support for full-text search queries and/or a > custom SQL function. > > E.g. Postgres syntax for FTS queries [1] is completely different to "LIKE" > operator. > > [1] > > https://www.postgresql.org/docs/9.5/textsearch-intro.html#TEXTSEARCH-MATCHING > > On Sat, Jul 24, 2021 at 4:49 PM Courtney Robinson < > courtney.robin...@hypi.io> > wrote: > > > Hey Ari, > > Yes, I wasn't suggesting that Solr should be used. That's just what we're > > doing now out of necessity. > > It was more the fact that Calcite's SqlOperator can be used to provide > the > > interface to Lucene. > > For all the reasons you mentioned and more, using Lucene is the right > > choice > > > > Calcite doesn't have support for Solr but it has an ES adapter which is > > what we modified to support Solr. > > > > Regards, > > Courtney Robinson > > Founder and CEO, Hypi > > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io> > > > > <https://hypi.io> > > https://hypi.io > > > > > > On Sat, Jul 24, 2021 at 1:59 PM Atri Sharma wrote: > > > > > What that entails is that the end user has to keep a Solr cluster > > running, > > > which comes with its own challenges (now you have to manage two systems > > > instead of one). > > > > > > I believe Calcite has native support for Solr? > > > > > > OTOH, having native Lucene indices allow us to control per partition > > > indices with no distributed overhead, since Lucene is a per node > instance > > > with no global coordination. > > > > > > On Sat, 24 Jul 2021, 16:57 Courtney Robinson, < > courtney.robin...@hypi.io > > > > > > wrote: > > > > > > > I'll add in here. > > > > I agree with you Valentin, the decoupled state of text queries makes > it > > > > useless for most use cases we have. > > > > > > > > As it relates to Calcite and Ignite 3, one approach (the one we're > > taking > > > > because we use calcite independent of Ignite) is to provide a bunch > of > > > SQL > > > > functions that we implement as SqlOperator > > > > < > > > > > > > > > > https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/SqlOperator.html > > > > >. > > > > I forget how we've done aggregation functions but we have those too > and > > > > they map to Solr aggregations (which ultimately end up in lucene). > > > > > > > > This allows Solr filters to take part in the rest of the query. It's > > > > probably more complex than this for Ignite but that's one possible > > route > > > > but we generate queries like select x from T0 where term(args to solr > > > term > > > > query) AND ... > > > > > > > > Regards, > > > > Courtney Robinson > > > > Founder and CEO, Hypi > > > > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io> > > > > > > > > <https://hypi.io> > > > > https://hypi.io > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:14 PM Valentin Kulichenko < > > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > > > Atri, > > > > > > > > > > Sure, go ahead. Let's put the ideas on paper and have a discussion. > > > > > > > > > > -Val > > > > > > > > > > On Fri, Jul 23, 2021 at
Re: Text Queries Support
What that entails is that the end user has to keep a Solr cluster running, which comes with its own challenges (now you have to manage two systems instead of one). I believe Calcite has native support for Solr? OTOH, having native Lucene indices allow us to control per partition indices with no distributed overhead, since Lucene is a per node instance with no global coordination. On Sat, 24 Jul 2021, 16:57 Courtney Robinson, wrote: > I'll add in here. > I agree with you Valentin, the decoupled state of text queries makes it > useless for most use cases we have. > > As it relates to Calcite and Ignite 3, one approach (the one we're taking > because we use calcite independent of Ignite) is to provide a bunch of SQL > functions that we implement as SqlOperator > < > https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/SqlOperator.html > >. > I forget how we've done aggregation functions but we have those too and > they map to Solr aggregations (which ultimately end up in lucene). > > This allows Solr filters to take part in the rest of the query. It's > probably more complex than this for Ignite but that's one possible route > but we generate queries like select x from T0 where term(args to solr term > query) AND ... > > Regards, > Courtney Robinson > Founder and CEO, Hypi > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io> > > <https://hypi.io> > https://hypi.io > > > On Fri, Jul 23, 2021 at 7:14 PM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Atri, > > > > Sure, go ahead. Let's put the ideas on paper and have a discussion. > > > > -Val > > > > On Fri, Jul 23, 2021 at 10:59 AM Atri Sharma wrote: > > > > > Thanks Andrey. > > > > > > I have collected answers or proposals to many of these questions and > > > would like to start a wiki page covering what we can do for Ignite 3. > > > > > > Does that sound good, please? > > > > > > On Fri, Jul 23, 2021 at 4:26 PM Andrey Mashenkov > > > wrote: > > > > > > > > Atri, > > > > > > > > First of all, I'd recommend going through the Ignite ticket to gather > > > > information about the current implementation issues and users' wants. > > > > Then look at a code to get a complete understanding of how things > work > > > now, > > > > which may help in future decisions. > > > > > > > > As we use the outdated Lucene version, some things may be irrelevant > > for > > > > the latest Lucene version. > > > > So, you will need expertise in the internals of modern Lucene version > > to > > > > understand what capabilities, guarantees, and limitations Lucene has > > and > > > > could bring to the Ignite. > > > > The expertise could be got from the Lucene project code or Lucene > > project > > > > dev-list. > > > > > > > > > > > > As for now, the potential capabilities are not clear to me. > > > > At first glance, I see the next topics that must be covered at first: > > > > > > > > General questions > > > > * How Lucene index can be split among the nodes? > > > > * If we'll have a single index for all partitions on the particular > > node, > > > > then how index records will be aware of partitioning? > > > > This is important to filter out backup records from the results to > > avoid > > > > duplicates. > > > > * How results from several nodes can be merged on the Reduce stage? > > > > * Does Lucene supports smth like JOIN operation or others that may > > > require > > > > data from another partition or index? > > > > If so, then it likes to multistep query with merging results on > > > > intermediate stages and requires detailed investigation and design. > > > > It is ok if Ignite will have some limitations here, but we would like > > to > > > > know about them at the early stage. > > > > * How effectively map Lucene files to the page memory? Is it even > > > possible? > > > > Otherwise, how to deal with potential OOM on large queries and memory > > > > capacity planning? > > > > > > > > Persistence. > > > > * How and what consistency guarantees could we have/expect? > > > > Seems, we may not be able to write physical records for Lucene index > to > > > our > > > > WAL. What can we do with this? > > > > > > >
Re: Text Queries Support
Thanks Andrey. I have collected answers or proposals to many of these questions and would like to start a wiki page covering what we can do for Ignite 3. Does that sound good, please? On Fri, Jul 23, 2021 at 4:26 PM Andrey Mashenkov wrote: > > Atri, > > First of all, I'd recommend going through the Ignite ticket to gather > information about the current implementation issues and users' wants. > Then look at a code to get a complete understanding of how things work now, > which may help in future decisions. > > As we use the outdated Lucene version, some things may be irrelevant for > the latest Lucene version. > So, you will need expertise in the internals of modern Lucene version to > understand what capabilities, guarantees, and limitations Lucene has and > could bring to the Ignite. > The expertise could be got from the Lucene project code or Lucene project > dev-list. > > > As for now, the potential capabilities are not clear to me. > At first glance, I see the next topics that must be covered at first: > > General questions > * How Lucene index can be split among the nodes? > * If we'll have a single index for all partitions on the particular node, > then how index records will be aware of partitioning? > This is important to filter out backup records from the results to avoid > duplicates. > * How results from several nodes can be merged on the Reduce stage? > * Does Lucene supports smth like JOIN operation or others that may require > data from another partition or index? > If so, then it likes to multistep query with merging results on > intermediate stages and requires detailed investigation and design. > It is ok if Ignite will have some limitations here, but we would like to > know about them at the early stage. > * How effectively map Lucene files to the page memory? Is it even possible? > Otherwise, how to deal with potential OOM on large queries and memory > capacity planning? > > Persistence. > * How and what consistency guarantees could we have/expect? > Seems, we may not be able to write physical records for Lucene index to our > WAL. What can we do with this? > > Transactions. > * Will we support transactions? > * Should Lucene be aware of Transaction and track mvcc (or whatever) > versions for the records? > * What will be consistency guarantees? > > UX > * How to add FullText search queries syntax into Calcite? > * AFAIK, the Lucene index has many properties for tuning. How will the user > configure the index? > * How and where to store the settings? What are cluster-wide and what a > local to the particular node? > * Will be all the settings immutable? Can be they changed on-fly? after > node/grid restart? > * Any limitations on query syntax? > > SQL > * Will we support FullText search in SQL? > * How to integrate Lucene index into Calcite? What is the cost model? > Splitting rules? Traits? > * What about consistency with DDL operations, e.g. column rename? > Ignite indices will operate column ID, so rename operation will not affect > the index. > > > With all of this, you can go with the IEP (or even some short summary) and > further POC and implementation. > That's a big deal, so let's discuss what could be done here. > > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma wrote: > > > I am actually happy to drive the feature for Ignite 3. FTS is very > > important for me and I think Ignite users will benefit from it > > greatly. > > > > If it makes sense to be focusing on Ignite 3 for this capability, I am > > eager to contribute there and lead the development. > > > > Please share your thoughts. > > > > On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov > > wrote: > > > > > > Hi Atri, > > > > > > All the Jira tickets we have on the Full-text search (FTS) thing are > > > targeted to Ignite 2. > > > > > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3, > > yet. > > > By the way, we are getting requests for this thing from the user side, > > and > > > definitely, > > > FTS would be a valuable feature for Ignite. > > > > > > It will be great if the one wants to drive it, any help will be > > appreciated. > > > > > > > > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma wrote: > > > > > > > Hello, > > > > > > > > An update, please. I am working through persistence of Lucene index > > using > > > > Ignite Dictionary, and will be asking some questions soon. > > > > > > > > I had one doubt - - where does this change go? Ignite 3? > > > > >
Re: Text Queries Support
The standard ways to deal with text based searches in SQL are the CONTAINS operator, the LIKE operator or specific functions (REGEXP_MATCHES, for eg). I do not think any of these are supported by Calcite at the moment. On Fri, Jul 23, 2021 at 11:20 PM Valentin Kulichenko wrote: > > In my experience, one of the biggest usability issues with the current > support of text queries is that they are completely decoupled from SQL. > I.e. you can either execute a SQL query OR a text query. Modern databases, > on the other hand, typically allow creating text-based indexes within > regular tables and then using those indexes within regular SQL queries. > Here is an example from Oracle: > https://docs.oracle.com/cd/B10501_01/text.920/a96517/cdefault.htm > > I believe this is something we can look into in the scope of Ignite 3. > Andrey, does Calcite have any support for this? What's your view on this? > > -Val > > On Fri, Jul 23, 2021 at 3:56 AM Andrey Mashenkov > wrote: > > > Atri, > > > > First of all, I'd recommend going through the Ignite ticket to gather > > information about the current implementation issues and users' wants. > > Then look at a code to get a complete understanding of how things work now, > > which may help in future decisions. > > > > As we use the outdated Lucene version, some things may be irrelevant for > > the latest Lucene version. > > So, you will need expertise in the internals of modern Lucene version to > > understand what capabilities, guarantees, and limitations Lucene has and > > could bring to the Ignite. > > The expertise could be got from the Lucene project code or Lucene project > > dev-list. > > > > > > As for now, the potential capabilities are not clear to me. > > At first glance, I see the next topics that must be covered at first: > > > > General questions > > * How Lucene index can be split among the nodes? > > * If we'll have a single index for all partitions on the particular node, > > then how index records will be aware of partitioning? > > This is important to filter out backup records from the results to avoid > > duplicates. > > * How results from several nodes can be merged on the Reduce stage? > > * Does Lucene supports smth like JOIN operation or others that may require > > data from another partition or index? > > If so, then it likes to multistep query with merging results on > > intermediate stages and requires detailed investigation and design. > > It is ok if Ignite will have some limitations here, but we would like to > > know about them at the early stage. > > * How effectively map Lucene files to the page memory? Is it even possible? > > Otherwise, how to deal with potential OOM on large queries and memory > > capacity planning? > > > > Persistence. > > * How and what consistency guarantees could we have/expect? > > Seems, we may not be able to write physical records for Lucene index to our > > WAL. What can we do with this? > > > > Transactions. > > * Will we support transactions? > > * Should Lucene be aware of Transaction and track mvcc (or whatever) > > versions for the records? > > * What will be consistency guarantees? > > > > UX > > * How to add FullText search queries syntax into Calcite? > > * AFAIK, the Lucene index has many properties for tuning. How will the user > > configure the index? > > * How and where to store the settings? What are cluster-wide and what a > > local to the particular node? > > * Will be all the settings immutable? Can be they changed on-fly? after > > node/grid restart? > > * Any limitations on query syntax? > > > > SQL > > * Will we support FullText search in SQL? > > * How to integrate Lucene index into Calcite? What is the cost model? > > Splitting rules? Traits? > > * What about consistency with DDL operations, e.g. column rename? > > Ignite indices will operate column ID, so rename operation will not affect > > the index. > > > > > > With all of this, you can go with the IEP (or even some short summary) and > > further POC and implementation. > > That's a big deal, so let's discuss what could be done here. > > > > On Fri, Jul 23, 2021 at 12:58 PM Atri Sharma wrote: > > > > > I am actually happy to drive the feature for Ignite 3. FTS is very > > > important for me and I think Ignite users will benefit from it > > > greatly. > > > > > > If it makes sense to be focusing on Ignite 3 for this capability, I am > > > eager to contribute there and lead the development. > > > >
Re: Text Queries Support
I am actually happy to drive the feature for Ignite 3. FTS is very important for me and I think Ignite users will benefit from it greatly. If it makes sense to be focusing on Ignite 3 for this capability, I am eager to contribute there and lead the development. Please share your thoughts. On Fri, Jul 23, 2021 at 3:21 PM Andrey Mashenkov wrote: > > Hi Atri, > > All the Jira tickets we have on the Full-text search (FTS) thing are > targeted to Ignite 2. > > AFAIK, we want, but we have NOT committed to FTS support in Ignite 3, yet. > By the way, we are getting requests for this thing from the user side, and > definitely, > FTS would be a valuable feature for Ignite. > > It will be great if the one wants to drive it, any help will be appreciated. > > > On Fri, Jul 23, 2021 at 12:12 PM Atri Sharma wrote: > > > Hello, > > > > An update, please. I am working through persistence of Lucene index using > > Ignite Dictionary, and will be asking some questions soon. > > > > I had one doubt - - where does this change go? Ignite 3? > > > > Also, I know we want to build native support for text searches in Ignite 3. > > Is the work I am proposing here part of that, or will that be a separate > > effort? > > > > On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev, > > wrote: > > > > > Hello! > > > > > > I think that number one is the most important one, then maybe it will see > > > more use and other deficiencies become more apparent, leading to more > > > tickets and visibility. > > > > > > Maybe 2. and 3. will even use a different approach when persistence is > > > implemented. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > пн, 28 июн. 2021 г. в 14:34, Atri Sharma : > > > > > > > Hello Again! > > > > > > > > I have been looking into the aforementioned and here are my follow up > > > > thoughts: > > > > > > > > 1. Support persistence of Lucene indexes. > > > > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of > > > > moving partitions first) > > > > 3. Figure out how to return scores from nodes and use them as sort > > > > parameters on the coordinator node > > > > (https://issues.apache.org/jira/browse/IGNITE-12291) > > > > > > > > Please let me know if this looks ok to make text queries functional? > > > > > > > > Atri > > > > > > > > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov > > > > wrote: > > > > > > > > > > Hi. > > > > > > > > > > One of the biggest issues with text queries is a lack of support for > > > > lucene > > > > > indices persistence, which makes this functionality useless if a > > > > > persistence is enabled. > > > > > > > > > > I would first take care of it. > > > > > > > > > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin > >: > > > > > > > > > > > Hi, Atri! > > > > > > > > > > > > You're right, Actually there is a lack of support for TextQueries. > > > For > > > > the > > > > > > last ticket I'm doing I see some obvious issues with them (no page > > > size > > > > > > support, for example). I'm glad that somebody wants to maintain > > this > > > > > > functionality. Thanks a lot! > > > > > > > > > > > > For the MergeSort algorithm there is already a patch for that [1]. > > > It's > > > > > > currently on review. This patch introduces an abstract reducer for > > > > > > CacheQueries with 2 implementations (unordered, merge-sort). Then > > > > TextQuery > > > > > > leverages on MergeSort to order results from multiple nodes by > > score. > > > > This > > > > > > patch also fixes the pageSize issue, I've mentioned before. Could > > you > > > > > > please check if it fully matches your idea? Any issues or comments > > > are > > > > > > welcome. > > > > > > > > > > > > I've prepared this ticket, because I need the MergeSort algorithm > > for > > > > the > > > > > > new type of queries I'm implementing (IndexQuery, it should also > > > > provide > > > > > >
Re: Text Queries Support
Hello, An update, please. I am working through persistence of Lucene index using Ignite Dictionary, and will be asking some questions soon. I had one doubt - - where does this change go? Ignite 3? Also, I know we want to build native support for text searches in Ignite 3. Is the work I am proposing here part of that, or will that be a separate effort? On Mon, 28 Jun 2021, 19:20 Ilya Kasnacheev, wrote: > Hello! > > I think that number one is the most important one, then maybe it will see > more use and other deficiencies become more apparent, leading to more > tickets and visibility. > > Maybe 2. and 3. will even use a different approach when persistence is > implemented. > > Regards, > -- > Ilya Kasnacheev > > > пн, 28 июн. 2021 г. в 14:34, Atri Sharma : > > > Hello Again! > > > > I have been looking into the aforementioned and here are my follow up > > thoughts: > > > > 1. Support persistence of Lucene indexes. > > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of > > moving partitions first) > > 3. Figure out how to return scores from nodes and use them as sort > > parameters on the coordinator node > > (https://issues.apache.org/jira/browse/IGNITE-12291) > > > > Please let me know if this looks ok to make text queries functional? > > > > Atri > > > > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov > > wrote: > > > > > > Hi. > > > > > > One of the biggest issues with text queries is a lack of support for > > lucene > > > indices persistence, which makes this functionality useless if a > > > persistence is enabled. > > > > > > I would first take care of it. > > > > > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin : > > > > > > > Hi, Atri! > > > > > > > > You're right, Actually there is a lack of support for TextQueries. > For > > the > > > > last ticket I'm doing I see some obvious issues with them (no page > size > > > > support, for example). I'm glad that somebody wants to maintain this > > > > functionality. Thanks a lot! > > > > > > > > For the MergeSort algorithm there is already a patch for that [1]. > It's > > > > currently on review. This patch introduces an abstract reducer for > > > > CacheQueries with 2 implementations (unordered, merge-sort). Then > > TextQuery > > > > leverages on MergeSort to order results from multiple nodes by score. > > This > > > > patch also fixes the pageSize issue, I've mentioned before. Could you > > > > please check if it fully matches your idea? Any issues or comments > are > > > > welcome. > > > > > > > > I've prepared this ticket, because I need the MergeSort algorithm for > > the > > > > new type of queries I'm implementing (IndexQuery, it should also > > provide > > > > ordered results over multiple nodes). Currently I'm not planning to > go > > > > further with TextQuery, so if you're going to support this it'll be a > > great > > > > contribution, I think. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703 > > > > [2] https://github.com/apache/ignite/pull/9081 > > > > > > > > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I have been looking into our text queries support and see that it > has > > > > > limited community support. > > > > > > > > > > Therefore, I volunteer to be the maintainer of the module and work > on > > > > > enhancing it further. > > > > > > > > > > First goal would be to move to Lucene 8.x, then work on sorted > reduce > > > > > - merge across nodes. Fundamentally, this is doable since Lucene > > ranks > > > > > documents according to their score, and documents are returned in > the > > > > > order of their score. Since the scoring function is homogeneous, > this > > > > > means that across nodes, we can compare scores and merge sort. > > > > > > > > > > Please let me know if I can take this up. > > > > > > > > > > Atri > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Atri > > > > > Apache Concerted > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > >
Re: Your Nabble Forum
Infra group is the ideal forum for this On Thu, 22 Jul 2021, 18:14 Denis Magda, wrote: > Dmitry, Ivan, > > It would be ideal if we can open up PonyMail archives for search engines. > It might be just one setting on a web server configuration. Dmitry, any > idea whom we should start discussing this with at ASF? > > -- > Denis > > - > Denis > > On Thu, Jul 22, 2021 at 3:32 PM Ivan Daschinsky > wrote: > > > +1 for pony mail. It is a great tool with nice interface with zero ads > > stuff. > > > > чт, 22 июл. 2021 г. в 15:26, Dmitry Pavlov : > > > > > Folks, > > > > > > My vote goes for ponymail. Actually it is a project under ASF itself, > it > > > is the part of Incubator. > > > > > > If we have a contributor who is aware about CEO setup, we can suggest > > help > > > and solve that issue for all ASF projects at once. > > > > > > https://github.com/apache/incubator-ponymail > > > https://ponymail.incubator.apache.org/support.html > > > https://lists.apache.org/list.html?us...@ponymail.apache.org > > > > > > I personaly use only lists.apache.org. And since at my dayjob gmail is > > > blocked, I also reply here. > > > > > > What do you think? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > On 2021/07/22 10:58:28, Denis Magda wrote: > > > > Probably, this is a reason why our Nabble's dev list forum got out of > > > sync. > > > > The latest message is dated as June 20th. > > > > > > > > Yeah, it's sad that ASF's mailing lists are not visible to search > > engines > > > > like Google. That's a big deal for those who search for a topic on > > search > > > > engines (99% of the human beings?). Let's think, I see three options > > > here: > > > > > > > >- Search for an alternate mail archiving solution that can be > > indexed > > > by > > > >Google > > > >- Talk to the ASF mates, probably there is a solution for user > lists > > > >- Give up the reins to StackOverflow that will naturally become a > > > >primary communication channel for the user-related questions. > > > > > > > > -- > > > > Denis > > > > > > > > - > > > > Denis > > > > > > > > On Thu, Jul 22, 2021 at 12:33 AM Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com> > > > > wrote: > > > > > > > > > Hello! > > > > > > > > > > They say they don't host mailing list archives anymore. I wonder > what > > > they > > > > > expect us to do here. > > > > > > > > > > I think it's a pity since Nabble had a great deal of SEO > visibility, > > > > > especially for user list. Unfortunately, Pony Mail is almost > useless > > in > > > > > that regard. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > ср, 21 июл. 2021 г. в 23:23, : > > > > > > > > > > > We are downsizing Nabble to one server. If you want to preserve > > your > > > > > > forum: > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > Then you should follow the instructions here: > > > > > > > > > > > > http://support.nabble.com/Downsizing-Nabble-tp7609715.html > > > > > > > > > > > > > > > > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > >
Re: MOVING Partitions and Rebalancing
Gentle ping. Please help with context here. On Mon, 5 Jul 2021, 13:03 Atri Sharma, wrote: > Hi All, > > As part of the text queries overhaul effort, I am looking into the > following ticket: > > https://issues.apache.org/jira/browse/IGNITE-12401 > > As I understand it, the problem lies in the fact that a partition can > move, thus causing duplicate data between two Lucene indices (on > different nodes). > > I wish to discuss the solution to this problem: > > 1. Fix the issue with MOVING partitions (need help here, since I am > not aware of the internals of rebalancing). > > 2. Circumvent the problem for text queries (maybe assign/use a > globally unique ID per entry, and use that to remove duplicates during > the merge phase?) > > Please share thoughts and inputs. > > Atri >
MOVING Partitions and Rebalancing
Hi All, As part of the text queries overhaul effort, I am looking into the following ticket: https://issues.apache.org/jira/browse/IGNITE-12401 As I understand it, the problem lies in the fact that a partition can move, thus causing duplicate data between two Lucene indices (on different nodes). I wish to discuss the solution to this problem: 1. Fix the issue with MOVING partitions (need help here, since I am not aware of the internals of rebalancing). 2. Circumvent the problem for text queries (maybe assign/use a globally unique ID per entry, and use that to remove duplicates during the merge phase?) Please share thoughts and inputs. Atri
Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output
Why not mask the default known sensitive options using a blocklist? On Thu, 1 Jul 2021, 22:24 Shishkov Ilya, wrote: > Folks, > > > Maybe we should add an extra JVM option (e.g. > IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default, > > but if set to 'true' then #ackVmOptions will print VM arguments even if > sensitive data is restricted? > > What do you think about an extra JVM option? > > чт, 1 июл. 2021 г. в 19:51, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Ivan, > > > > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths > > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data > > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to > be > > sensitive. > > > > Once again - I would exclude any property that can contain user-specific > > information. Only our internal settings (stuff > > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs. > > > > -Val > > > > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky > > wrote: > > > > > We can add add an extra param in annotation, that blocks param to be > > > printed, just set it to false by default and block it wheb set to true > > > > > > чт, 1 июл. 2021 г., 19:45 Atri Sharma : > > > > > > > What if we allowed a blocklist of parameters that are never printed? > > > > > > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, < > > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > > > Not all of them are OK to be printed out. At the very least, we > > should > > > > have > > > > > a mechanism to exclude some of them. I would still go with opt-in > > > rather > > > > > than opt-out though, but I guess that is up to a discussion. > > > > > > > > > > -Val > > > > > > > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky < > ivanda...@gmail.com> > > > > > wrote: > > > > > > > > > > > This is security through obscurity, an obvious and a well-known > > anti > > > > > > pattern. I suppose that printing jvm options, that is registered > by > > > > > > @IgniteSystemProperty annotation is an ideal variant > > > > > > > > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko < > > > > > > valentin.kuliche...@gmail.com > > > > > > >: > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > *Anything* that a user provides to the system can potentially > be > > > > > > considered > > > > > > > sensitive information. This includes the VM arguments. We can't > > > > predict > > > > > > > what exactly one can put there, so let's not make assumptions. > > > > > > > > > > > > > > When dealing with security, we should be as conservative as > > > possible. > > > > > > That > > > > > > > said, I do not even agree with the pattern approach - there > might > > > be > > > > a > > > > > > > user's system property named IGNITE_xxx. It is also possible > for > > > our > > > > > > > internal properties to contain sensitive information (not all > of > > > them > > > > > are > > > > > > > boolean flags). > > > > > > > > > > > > > > The only option I see is to print out specific properties for > > which > > > > we > > > > > > > agree that they are safe. For example, we can introduce an > > > annotation > > > > > > that > > > > > > > would mark "safe" properties in IgniteSystemProperties; we will > > > then > > > > > > print > > > > > > > out only those that are marked with the annotation. > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин < > > > > > > slava.kopti...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Ivan, > > > > >
Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output
What if we allowed a blocklist of parameters that are never printed? On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, < valentin.kuliche...@gmail.com> wrote: > Not all of them are OK to be printed out. At the very least, we should have > a mechanism to exclude some of them. I would still go with opt-in rather > than opt-out though, but I guess that is up to a discussion. > > -Val > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky > wrote: > > > This is security through obscurity, an obvious and a well-known anti > > pattern. I suppose that printing jvm options, that is registered by > > @IgniteSystemProperty annotation is an ideal variant > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko < > > valentin.kuliche...@gmail.com > > >: > > > > > Folks, > > > > > > *Anything* that a user provides to the system can potentially be > > considered > > > sensitive information. This includes the VM arguments. We can't predict > > > what exactly one can put there, so let's not make assumptions. > > > > > > When dealing with security, we should be as conservative as possible. > > That > > > said, I do not even agree with the pattern approach - there might be a > > > user's system property named IGNITE_xxx. It is also possible for our > > > internal properties to contain sensitive information (not all of them > are > > > boolean flags). > > > > > > The only option I see is to print out specific properties for which we > > > agree that they are safe. For example, we can introduce an annotation > > that > > > would mark "safe" properties in IgniteSystemProperties; we will then > > print > > > out only those that are marked with the annotation. > > > > > > -Val > > > > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин < > > slava.kopti...@gmail.com > > > > > > > wrote: > > > > > > > Hello Ivan, > > > > > > > > > At least, we could just hide params that match a specific pattern > > > > Yes, we can filter out all vm options that do not relate to Ignite, > for > > > > instance. > > > > > > > > > Ilya, go ahead, file ticket and prepare a PR. > > > > Please do not rush. Let's listen to other community members. This > > > question > > > > is about security and it should not be discussed in a hurry (even > > though > > > it > > > > looks like an obvious thing). > > > > > > > > Thanks, > > > > S. > > > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky : > > > > > > > > > I suppose, that all normal users should not suffer from this > > > > restrictions. > > > > > Nobody will pass password using jvm options. It is absolutely > insane, > > > > > normal users pass passwords using environment variables. > > > > > > > > > > At least, we could just hide params that match specific pattern > > > > > > > > > > Ilya, go ahead, file ticket and prepare a PR. > > > > > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин < > > slava.kopti...@gmail.com > > > >: > > > > > > > > > > > Hello, > > > > > > > > > > > > Unfortunately, the user can pass its own system properties via > JVM > > > > > options > > > > > > as follows: -DMY_SECRET_PASSWORD=123 > > > > > > It does not seem, this approach is the best one. However, the > user > > > > should > > > > > > have a "kostyl" in order to hide these properties and values in > the > > > log > > > > > > file, IMHO. > > > > > > > > > > > > Thanks, > > > > > > S. > > > > > > > > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya < > shishkovi...@gmail.com > > >: > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > > > This feature [1, 2] prevents logging of the VM arguments when > > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till > > > now, > > > > > > method > > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3]. > > > > > > > > > > > > > > Is this behaviour actual now? Often, we should be able to get > > from > > > > logs > > > > > > the > > > > > > > actual VM options used at startup even if output of sensitive > > data > > > is > > > > > > > restricted. > > > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991 > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93 > > > > > > > 3. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002 > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Re[2]: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output
AFAIK, this allows users to pass in custom VM options which may be undesirable. On Thu, Jul 1, 2021 at 12:07 PM Zhenya Stanilovsky wrote: > > > +1 for reverting, can anybody (possibly ticket starter?) explain how jvm > settings will rise some security problems ? > > > > >I suppose, that we should revert this particular line. I don't understand > >who ever considers vm options as sensitive info. > > > >ср, 30 июн. 2021 г., 22:52 Shishkov Ilya < shishkovi...@gmail.com >: > > > >> Hi Igniters, > >> > >> This feature [1, 2] prevents logging of the VM arguments when > >> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method > >> IgniteKernal#ackVmArguments remains mostly the same [3]. > >> > >> Is this behaviour actual now? Often, we should be able to get from logs the > >> actual VM options used at startup even if output of sensitive data is > >> restricted. > >> > >> 1. https://issues.apache.org/jira/browse/IGNITE-4991 > >> 2. > >> > >> > >> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93 > >> 3. > >> > >> > >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002 > >> > > > > -- Regards, Atri Apache Concerted
Re: Collision SPI Not Adhering to Specification
I have opened a JIRA for the same: https://issues.apache.org/jira/browse/IGNITE-15043 Unless objections, I plan on sketching an implementation plan. On Thu, Jul 1, 2021 at 1:07 AM Atri Sharma wrote: > > Hi All, > > I have been playing around with Collision SPI and specifically used > FifoQueueCollisionSPI and noticed that it is not really adhering to > the specified task of restricting the number of concurrent tasks that > can be run. > > Specifically, if there is only one slot available and N tasks > concurrently land onto the node, all nodes will take the current > active count, compare it with maximum jobs count and proceed. > > Essentially, we have no concurrency safety there. > > I propose refactoring FifoQueueCollisionSPI to use semaphores and/or > atomic variables to ensure that all jobs get a realistic view of the > active count. > > Please share your thoughts, > > Regards, > > Atri > > -- > Regards, > > Atri > Apache Concerted -- Regards, Atri Apache Concerted
Collision SPI Not Adhering to Specification
Hi All, I have been playing around with Collision SPI and specifically used FifoQueueCollisionSPI and noticed that it is not really adhering to the specified task of restricting the number of concurrent tasks that can be run. Specifically, if there is only one slot available and N tasks concurrently land onto the node, all nodes will take the current active count, compare it with maximum jobs count and proceed. Essentially, we have no concurrency safety there. I propose refactoring FifoQueueCollisionSPI to use semaphores and/or atomic variables to ensure that all jobs get a realistic view of the active count. Please share your thoughts, Regards, Atri -- Regards, Atri Apache Concerted
Re: Publishing Spring Sessions to Maven Repo
Gentle ping On Thu, Jun 24, 2021 at 6:36 PM Atri Sharma wrote: > > Hi All, > > Can we please publish Spring Sessions Ext artifacts to the Maven repo? > > Atri -- Regards, Atri Apache Concerted
Re: Text Queries Support
Hello Again! I have been looking into the aforementioned and here are my follow up thoughts: 1. Support persistence of Lucene indexes. 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of moving partitions first) 3. Figure out how to return scores from nodes and use them as sort parameters on the coordinator node (https://issues.apache.org/jira/browse/IGNITE-12291) Please let me know if this looks ok to make text queries functional? Atri On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov wrote: > > Hi. > > One of the biggest issues with text queries is a lack of support for lucene > indices persistence, which makes this functionality useless if a > persistence is enabled. > > I would first take care of it. > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin : > > > Hi, Atri! > > > > You're right, Actually there is a lack of support for TextQueries. For the > > last ticket I'm doing I see some obvious issues with them (no page size > > support, for example). I'm glad that somebody wants to maintain this > > functionality. Thanks a lot! > > > > For the MergeSort algorithm there is already a patch for that [1]. It's > > currently on review. This patch introduces an abstract reducer for > > CacheQueries with 2 implementations (unordered, merge-sort). Then TextQuery > > leverages on MergeSort to order results from multiple nodes by score. This > > patch also fixes the pageSize issue, I've mentioned before. Could you > > please check if it fully matches your idea? Any issues or comments are > > welcome. > > > > I've prepared this ticket, because I need the MergeSort algorithm for the > > new type of queries I'm implementing (IndexQuery, it should also provide > > ordered results over multiple nodes). Currently I'm not planning to go > > further with TextQuery, so if you're going to support this it'll be a great > > contribution, I think. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703 > > [2] https://github.com/apache/ignite/pull/9081 > > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma wrote: > > > > > Hi All, > > > > > > I have been looking into our text queries support and see that it has > > > limited community support. > > > > > > Therefore, I volunteer to be the maintainer of the module and work on > > > enhancing it further. > > > > > > First goal would be to move to Lucene 8.x, then work on sorted reduce > > > - merge across nodes. Fundamentally, this is doable since Lucene ranks > > > documents according to their score, and documents are returned in the > > > order of their score. Since the scoring function is homogeneous, this > > > means that across nodes, we can compare scores and merge sort. > > > > > > Please let me know if I can take this up. > > > > > > Atri > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > > > > -- > > Best regards, > Alexei Scherbakov -- Regards, Atri Apache Concerted
Publishing Spring Sessions to Maven Repo
Hi All, Can we please publish Spring Sessions Ext artifacts to the Maven repo? Atri
Re: Spring Sessions With Ignite As Backing Store
Hello all, Sam has approved the PR and tests are passing: https://ci.ignite.apache.org/buildConfiguration/IgniteExtensions_Tests_SpringSession/6055833 Can we please approve and merge? On Fri, 18 Jun 2021, 13:28 Atri Sharma, wrote: > Hi All, > > Please find PR for the aforementioned issue: > > https://github.com/apache/ignite-extensions/pull/63 > > @Данилов Семён Please help in reviewing. > > Regards, > > Atri > > -- > Regards, > > Atri > Apache Concerted >
Re: Text Queries Support
Thank you, Maksin and Alexei! To dive a bit deeper, what are our biggest issues with text queries today? One is persistence, the other (IIUC) is the fact that we cannot order results from different nodes (which PR 9081 seems to have resolved?) What else would be pending for text queries to become usable? Atri On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov wrote: > > Hi. > > One of the biggest issues with text queries is a lack of support for lucene > indices persistence, which makes this functionality useless if a > persistence is enabled. > > I would first take care of it. > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin : > > > Hi, Atri! > > > > You're right, Actually there is a lack of support for TextQueries. For the > > last ticket I'm doing I see some obvious issues with them (no page size > > support, for example). I'm glad that somebody wants to maintain this > > functionality. Thanks a lot! > > > > For the MergeSort algorithm there is already a patch for that [1]. It's > > currently on review. This patch introduces an abstract reducer for > > CacheQueries with 2 implementations (unordered, merge-sort). Then TextQuery > > leverages on MergeSort to order results from multiple nodes by score. This > > patch also fixes the pageSize issue, I've mentioned before. Could you > > please check if it fully matches your idea? Any issues or comments are > > welcome. > > > > I've prepared this ticket, because I need the MergeSort algorithm for the > > new type of queries I'm implementing (IndexQuery, it should also provide > > ordered results over multiple nodes). Currently I'm not planning to go > > further with TextQuery, so if you're going to support this it'll be a great > > contribution, I think. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703 > > [2] https://github.com/apache/ignite/pull/9081 > > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma wrote: > > > > > Hi All, > > > > > > I have been looking into our text queries support and see that it has > > > limited community support. > > > > > > Therefore, I volunteer to be the maintainer of the module and work on > > > enhancing it further. > > > > > > First goal would be to move to Lucene 8.x, then work on sorted reduce > > > - merge across nodes. Fundamentally, this is doable since Lucene ranks > > > documents according to their score, and documents are returned in the > > > order of their score. Since the scoring function is homogeneous, this > > > means that across nodes, we can compare scores and merge sort. > > > > > > Please let me know if I can take this up. > > > > > > Atri > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > > > > -- > > Best regards, > Alexei Scherbakov -- Regards, Atri Apache Concerted
Text Queries Support
Hi All, I have been looking into our text queries support and see that it has limited community support. Therefore, I volunteer to be the maintainer of the module and work on enhancing it further. First goal would be to move to Lucene 8.x, then work on sorted reduce - merge across nodes. Fundamentally, this is doable since Lucene ranks documents according to their score, and documents are returned in the order of their score. Since the scoring function is homogeneous, this means that across nodes, we can compare scores and merge sort. Please let me know if I can take this up. Atri -- Regards, Atri Apache Concerted
Spring Sessions With Ignite As Backing Store
Hi All, Please find PR for the aforementioned issue: https://github.com/apache/ignite-extensions/pull/63 @Данилов Семён Please help in reviewing. Regards, Atri -- Regards, Atri Apache Concerted
Re: Change IGNITE_PDS_WAL_REBALANCE_THRESHOLD from System property to Configuraton
+1 On Sun, 9 May 2021, 21:33 Stanislav Lukyanov, wrote: > Hi Eduard, > > I strongly believe that if a configuration option is cluster wide then it > belongs to distributed metastore and not to IgniteConfiguration. > This allows to get cluster-wide consistency guarantees and API for dynamic > change out of the box (need to teach the internals to re-read the property > from DMS every time of course). > > WDYT? > > Stan > > > On 6 May 2021, at 16:35, Eduard Rakhmankulov > wrote: > > > > Some addition. > > > > I want to add configuration to > > > org.apache.ignite.configuration.DataStorageConfiguration#getDefaultWarmUpConfiguration#getP > > artitionWalRebalanceThreshold > > which will have same semantics as system property (number of entries in > WAL > > to trigger rebalance). > > > > On Thu, 6 May 2021 at 15:50, Eduard Rakhmankulov > > wrote: > > > >> Hello, Igniters! > >> > >> I suggest changing IGNITE_PDS_WAL_REBALANCE_THRESHOLD from system > >> properties to IgniteConfiguration. > >> This configuration is effectively cluster-wide (because only the > >> coordinator's configuration matters when the heuristic with this > property > >> applies). > >> > >> It is easier to validate that we have the same configuration on all > nodes > >> than system property (in the case when another coordinator was elected). > >> > >> -- > >> Best regards, Eduard. > >> > > > > > > -- > > С уважением, Рахманкулов Э.Р. > >
Re: IP Filtering in IPFinders
Thank you, Val. Ilya, please let me know if the PR looks ok. On Thu, 29 Apr 2021, 00:19 Valentin Kulichenko, < valentin.kuliche...@gmail.com> wrote: > I'm OK with the design. > > Ilya, please feel free to merge if the implementation and tests look good > to you. > > -Val > > On Wed, Apr 28, 2021 at 1:07 AM Atri Sharma wrote: > > > Hi Ilya and Val, > > > > Thank you for taking a look and providing insights. I have updated the > > PR and raised another iteration. > > > > Val, I have moved the configuration to TcpDiscoverySpi. > > > > Please see and let me know your thoughts and comments. > > > > Regards, > > > > Atri > > > > On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko > > wrote: > > > > > > Hi Atri, > > > > > > I've noticed that you added the property to the IgniteConfiguration, > but > > > it's applied only within the discovery. I feel like something is wrong > > here. > > > > > > If this feature only relates to the discovery, then we should have the > > > configuration property on the TcpDiscoverySpi instead. Otherwise, the > > > filtering should be applied to all network components (although I'm not > > > necessarily sure what that would imply). > > > > > > What do you think? > > > > > > -Val > > > > > > On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma wrote: > > > > > > > Hi Val and Ilya, > > > > > > > > Thank you for taking the time to pursue this issue. > > > > > > > > I have raised a new PR for the discussed approach. Please see and let > > > > me know what you think: > > > > > > > > https://github.com/apache/ignite/pull/9048 > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev > > > > wrote: > > > > > > > > > > Hello! > > > > > > > > > > I'm still not fully convinced, but Val's approach sounds rational > to > > me. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma : > > > > > > > > > > > Hello! > > > > > > > > > > > > I actually saw the shared container scenario being tried by > > somebody > > > > > > who wanted an external script to monitor all IPs being used by > his > > > > > > clusters and hence thought of this idea. Another thing that came > in > > > > > > was the Firewall blocking a few IP addresses, hence the idea. > > > > > > > > > > > > I feel that the footprint of this change is small, and can be > > useful > > > > > > for esoteric use cases too without really interfering in any > > existing > > > > > > code path. Val's suggestion seems the right way to go since it > > gives > > > > > > the functionality without much change. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev > > > > > > wrote: > > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > AFAIK, a S3 container, Azure blob container, etc, is a > relatively > > > > > > > lightweight entity, similar to a table in an SQL database. Why > > would > > > > > > > different clusters need to share the same discovery storage > > > > container? > > > > > > > When I tested Azure IP finder, it created several blob > containers > > > > for me > > > > > > on > > > > > > > demand, based on the parameter passed to IP finder. If I wanted > > to > > > > have > > > > > > > more than one cluster it should have been seamless already. > > > > > > > > > > > > > > I can theoretically see how address filtering may be useful to > > remove > > > > > > > public / private addresses or Docker gateway address, but it is > > > > usually > > > > > > > handled by setting localHost setting, although requiring tuning > > it &g
Re: IP Filtering in IPFinders
Hi Ilya and Val, Thank you for taking a look and providing insights. I have updated the PR and raised another iteration. Val, I have moved the configuration to TcpDiscoverySpi. Please see and let me know your thoughts and comments. Regards, Atri On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko wrote: > > Hi Atri, > > I've noticed that you added the property to the IgniteConfiguration, but > it's applied only within the discovery. I feel like something is wrong here. > > If this feature only relates to the discovery, then we should have the > configuration property on the TcpDiscoverySpi instead. Otherwise, the > filtering should be applied to all network components (although I'm not > necessarily sure what that would imply). > > What do you think? > > -Val > > On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma wrote: > > > Hi Val and Ilya, > > > > Thank you for taking the time to pursue this issue. > > > > I have raised a new PR for the discussed approach. Please see and let > > me know what you think: > > > > https://github.com/apache/ignite/pull/9048 > > > > Regards, > > > > Atri > > > > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > I'm still not fully convinced, but Val's approach sounds rational to me. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma : > > > > > > > Hello! > > > > > > > > I actually saw the shared container scenario being tried by somebody > > > > who wanted an external script to monitor all IPs being used by his > > > > clusters and hence thought of this idea. Another thing that came in > > > > was the Firewall blocking a few IP addresses, hence the idea. > > > > > > > > I feel that the footprint of this change is small, and can be useful > > > > for esoteric use cases too without really interfering in any existing > > > > code path. Val's suggestion seems the right way to go since it gives > > > > the functionality without much change. > > > > > > > > Thoughts? > > > > > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev > > > > wrote: > > > > > > > > > > Hello! > > > > > > > > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively > > > > > lightweight entity, similar to a table in an SQL database. Why would > > > > > different clusters need to share the same discovery storage > > container? > > > > > When I tested Azure IP finder, it created several blob containers > > for me > > > > on > > > > > demand, based on the parameter passed to IP finder. If I wanted to > > have > > > > > more than one cluster it should have been seamless already. > > > > > > > > > > I can theoretically see how address filtering may be useful to remove > > > > > public / private addresses or Docker gateway address, but it is > > usually > > > > > handled by setting localHost setting, although requiring tuning it > > for > > > > each > > > > > instance individually. Overall benefit seems to small. > > > > > > > > > > This is why I am asking, do you have any specific scenario in mind > > where > > > > > this feature is an enabler? How did you arrive at the conclusion to > > go > > > > > forward with it? > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma : > > > > > > > > > > > Hi Val, > > > > > > > > > > > > Consider a scenario where multiple Ignite clusters are running and > > for > > > > > > operational ease (and also compliance, in some cases, e.g. to make > > > > > > auditing easier), people can configure cloud based IP finders to > > share > > > > > > the same container (blob container in Azure, S3 container in AWS > > etc). > > > > > > > > > > > > In such a case, IPs for all clusters will be in the same container. > > > > > > IPFinders of both the clusters will read the entire list. In this > > > > > > case,
Re: IP Filtering in IPFinders
Hi Val and Ilya, Thank you for taking the time to pursue this issue. I have raised a new PR for the discussed approach. Please see and let me know what you think: https://github.com/apache/ignite/pull/9048 Regards, Atri On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev wrote: > > Hello! > > I'm still not fully convinced, but Val's approach sounds rational to me. > > Regards, > -- > Ilya Kasnacheev > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma : > > > Hello! > > > > I actually saw the shared container scenario being tried by somebody > > who wanted an external script to monitor all IPs being used by his > > clusters and hence thought of this idea. Another thing that came in > > was the Firewall blocking a few IP addresses, hence the idea. > > > > I feel that the footprint of this change is small, and can be useful > > for esoteric use cases too without really interfering in any existing > > code path. Val's suggestion seems the right way to go since it gives > > the functionality without much change. > > > > Thoughts? > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively > > > lightweight entity, similar to a table in an SQL database. Why would > > > different clusters need to share the same discovery storage container? > > > When I tested Azure IP finder, it created several blob containers for me > > on > > > demand, based on the parameter passed to IP finder. If I wanted to have > > > more than one cluster it should have been seamless already. > > > > > > I can theoretically see how address filtering may be useful to remove > > > public / private addresses or Docker gateway address, but it is usually > > > handled by setting localHost setting, although requiring tuning it for > > each > > > instance individually. Overall benefit seems to small. > > > > > > This is why I am asking, do you have any specific scenario in mind where > > > this feature is an enabler? How did you arrive at the conclusion to go > > > forward with it? > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma : > > > > > > > Hi Val, > > > > > > > > Consider a scenario where multiple Ignite clusters are running and for > > > > operational ease (and also compliance, in some cases, e.g. to make > > > > auditing easier), people can configure cloud based IP finders to share > > > > the same container (blob container in Azure, S3 container in AWS etc). > > > > > > > > In such a case, IPs for all clusters will be in the same container. > > > > IPFinders of both the clusters will read the entire list. In this > > > > case, address filtering will help ignore the irrelevant IP addresses. > > > > > > > > Thank you for pointing me to the alternate direction. Let me research > > > > that and revert. > > > > > > > > Atri > > > > > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko > > > > wrote: > > > > > > > > > > Hi Atri, > > > > > > > > > > Can you describe the scenario in a little more detail? What exactly > > do > > > > you > > > > > mean by a container shared by multiple clusters? What are the > > > > consequences > > > > > of this? How does the proposed solution solve the problem? > > > > > > > > > > Also, I would suggest revisiting the design - I'm not sure such > > filtering > > > > > should be done on the IP finder level. Why not do this on the SPI > > level > > > > > instead? I would simply add something like "addressFilter" to the > > > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you > > will > > > > > be able to provide any implementations, including regex or anything > > else. > > > > > > > > > > -Val > > > > > > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > When a container is shared by multiple clusters, then this can be > > > > useful > > > > > > for filtering IPs. > > > >
Re: IP Filtering in IPFinders
Hello! I actually saw the shared container scenario being tried by somebody who wanted an external script to monitor all IPs being used by his clusters and hence thought of this idea. Another thing that came in was the Firewall blocking a few IP addresses, hence the idea. I feel that the footprint of this change is small, and can be useful for esoteric use cases too without really interfering in any existing code path. Val's suggestion seems the right way to go since it gives the functionality without much change. Thoughts? On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev wrote: > > Hello! > > AFAIK, a S3 container, Azure blob container, etc, is a relatively > lightweight entity, similar to a table in an SQL database. Why would > different clusters need to share the same discovery storage container? > When I tested Azure IP finder, it created several blob containers for me on > demand, based on the parameter passed to IP finder. If I wanted to have > more than one cluster it should have been seamless already. > > I can theoretically see how address filtering may be useful to remove > public / private addresses or Docker gateway address, but it is usually > handled by setting localHost setting, although requiring tuning it for each > instance individually. Overall benefit seems to small. > > This is why I am asking, do you have any specific scenario in mind where > this feature is an enabler? How did you arrive at the conclusion to go > forward with it? > > Regards, > -- > Ilya Kasnacheev > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma : > > > Hi Val, > > > > Consider a scenario where multiple Ignite clusters are running and for > > operational ease (and also compliance, in some cases, e.g. to make > > auditing easier), people can configure cloud based IP finders to share > > the same container (blob container in Azure, S3 container in AWS etc). > > > > In such a case, IPs for all clusters will be in the same container. > > IPFinders of both the clusters will read the entire list. In this > > case, address filtering will help ignore the irrelevant IP addresses. > > > > Thank you for pointing me to the alternate direction. Let me research > > that and revert. > > > > Atri > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko > > wrote: > > > > > > Hi Atri, > > > > > > Can you describe the scenario in a little more detail? What exactly do > > you > > > mean by a container shared by multiple clusters? What are the > > consequences > > > of this? How does the proposed solution solve the problem? > > > > > > Also, I would suggest revisiting the design - I'm not sure such filtering > > > should be done on the IP finder level. Why not do this on the SPI level > > > instead? I would simply add something like "addressFilter" to the > > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will > > > be able to provide any implementations, including regex or anything else. > > > > > > -Val > > > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma wrote: > > > > > > > Hi, > > > > > > > > When a container is shared by multiple clusters, then this can be > > useful > > > > for filtering IPs. > > > > > > > > Also, things like VPC based barriers can be circumvented using this > > > > technique. > > > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, > > > > > > wrote: > > > > > > > > > Hello! > > > > > > > > > > What are the expected use cases for this feature? Can you please > > > > elaborate? > > > > > > > > > > Thanks, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma : > > > > > > > > > > > Hi All, > > > > > > > > > > > > I have opened the following JIRA for the said topic: > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606 > > > > > > > > > > > > The concept is to filter IPs based on a pattern or a blocklist in > > > > > > IPFinders while consuming IPs. This is more pertinent for cloud > > based > > > > > > IPFinders since they can have shared containers. > > > > > > > > > > > > For the moment, I have implemented regex based filtering: > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607 > > > > > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the > > same to > > > > > > other IP finders. > > > > > > > > > > > > Please see the PR: > > > > > > > > > > > > https://github.com/apache/ignite/pull/9024 > > > > > > > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > Apache Concerted > > > > > > > > > > > > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: IP Filtering in IPFinders
Another thing is IP addresses blocked by firewalls -- such IPs will cause the cluster bootstrap to slow down. On Thu, Apr 22, 2021 at 10:20 AM Atri Sharma wrote: > > Hi Val, > > Consider a scenario where multiple Ignite clusters are running and for > operational ease (and also compliance, in some cases, e.g. to make > auditing easier), people can configure cloud based IP finders to share > the same container (blob container in Azure, S3 container in AWS etc). > > In such a case, IPs for all clusters will be in the same container. > IPFinders of both the clusters will read the entire list. In this > case, address filtering will help ignore the irrelevant IP addresses. > > Thank you for pointing me to the alternate direction. Let me research > that and revert. > > Atri > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko > wrote: > > > > Hi Atri, > > > > Can you describe the scenario in a little more detail? What exactly do you > > mean by a container shared by multiple clusters? What are the consequences > > of this? How does the proposed solution solve the problem? > > > > Also, I would suggest revisiting the design - I'm not sure such filtering > > should be done on the IP finder level. Why not do this on the SPI level > > instead? I would simply add something like "addressFilter" to the > > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will > > be able to provide any implementations, including regex or anything else. > > > > -Val > > > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma wrote: > > > > > Hi, > > > > > > When a container is shared by multiple clusters, then this can be useful > > > for filtering IPs. > > > > > > Also, things like VPC based barriers can be circumvented using this > > > technique. > > > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, > > > wrote: > > > > > > > Hello! > > > > > > > > What are the expected use cases for this feature? Can you please > > > elaborate? > > > > > > > > Thanks, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma : > > > > > > > > > Hi All, > > > > > > > > > > I have opened the following JIRA for the said topic: > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606 > > > > > > > > > > The concept is to filter IPs based on a pattern or a blocklist in > > > > > IPFinders while consuming IPs. This is more pertinent for cloud based > > > > > IPFinders since they can have shared containers. > > > > > > > > > > For the moment, I have implemented regex based filtering: > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607 > > > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to > > > > > other IP finders. > > > > > > > > > > Please see the PR: > > > > > > > > > > https://github.com/apache/ignite/pull/9024 > > > > > > > > > > Regards, > > > > > > > > > > Atri > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Atri > > > > > Apache Concerted > > > > > > > > > > > > > > -- > Regards, > > Atri > Apache Concerted -- Regards, Atri Apache Concerted
Re: IP Filtering in IPFinders
Hi Andrey, Thanks for the message. Yes, that is the case since I wanted to show the functionality to the group first. Val has made a point about an alternate approach which seems cleaner to me, so let me explore that. If we go that route, then we will not need to change at IPFinder level which will make the change cleaner. Atri On Wed, Apr 21, 2021 at 10:50 PM Andrey Mashenkov wrote: > > Hi Atri, > > You've added a new property to a base TcpDiscoveryIpFinder interface. > Actually, the only Azure IpFinder uses this setting, but the others. > This behavior may confuse the users. > > Would you mind either making regexp filter setting a part of Azure IpFinder > only or fix other IpFinders as well? > > > On Wed, Apr 21, 2021 at 7:04 PM Atri Sharma wrote: > > > Hi, > > > > When a container is shared by multiple clusters, then this can be useful > > for filtering IPs. > > > > Also, things like VPC based barriers can be circumvented using this > > technique. > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, > > wrote: > > > > > Hello! > > > > > > What are the expected use cases for this feature? Can you please > > elaborate? > > > > > > Thanks, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma : > > > > > > > Hi All, > > > > > > > > I have opened the following JIRA for the said topic: > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606 > > > > > > > > The concept is to filter IPs based on a pattern or a blocklist in > > > > IPFinders while consuming IPs. This is more pertinent for cloud based > > > > IPFinders since they can have shared containers. > > > > > > > > For the moment, I have implemented regex based filtering: > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607 > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to > > > > other IP finders. > > > > > > > > Please see the PR: > > > > > > > > https://github.com/apache/ignite/pull/9024 > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > > > > -- > Best regards, > Andrey V. Mashenkov -- Regards, Atri Apache Concerted
Re: IP Filtering in IPFinders
Hi Val, Consider a scenario where multiple Ignite clusters are running and for operational ease (and also compliance, in some cases, e.g. to make auditing easier), people can configure cloud based IP finders to share the same container (blob container in Azure, S3 container in AWS etc). In such a case, IPs for all clusters will be in the same container. IPFinders of both the clusters will read the entire list. In this case, address filtering will help ignore the irrelevant IP addresses. Thank you for pointing me to the alternate direction. Let me research that and revert. Atri On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko wrote: > > Hi Atri, > > Can you describe the scenario in a little more detail? What exactly do you > mean by a container shared by multiple clusters? What are the consequences > of this? How does the proposed solution solve the problem? > > Also, I would suggest revisiting the design - I'm not sure such filtering > should be done on the IP finder level. Why not do this on the SPI level > instead? I would simply add something like "addressFilter" to the > TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will > be able to provide any implementations, including regex or anything else. > > -Val > > On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma wrote: > > > Hi, > > > > When a container is shared by multiple clusters, then this can be useful > > for filtering IPs. > > > > Also, things like VPC based barriers can be circumvented using this > > technique. > > > > On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, > > wrote: > > > > > Hello! > > > > > > What are the expected use cases for this feature? Can you please > > elaborate? > > > > > > Thanks, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma : > > > > > > > Hi All, > > > > > > > > I have opened the following JIRA for the said topic: > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14606 > > > > > > > > The concept is to filter IPs based on a pattern or a blocklist in > > > > IPFinders while consuming IPs. This is more pertinent for cloud based > > > > IPFinders since they can have shared containers. > > > > > > > > For the moment, I have implemented regex based filtering: > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-14607 > > > > > > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to > > > > other IP finders. > > > > > > > > Please see the PR: > > > > > > > > https://github.com/apache/ignite/pull/9024 > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > -- Regards, Atri Apache Concerted
Re: IP Filtering in IPFinders
Hi, When a container is shared by multiple clusters, then this can be useful for filtering IPs. Also, things like VPC based barriers can be circumvented using this technique. On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, wrote: > Hello! > > What are the expected use cases for this feature? Can you please elaborate? > > Thanks, > -- > Ilya Kasnacheev > > > ср, 21 апр. 2021 г. в 08:23, Atri Sharma : > > > Hi All, > > > > I have opened the following JIRA for the said topic: > > > > https://issues.apache.org/jira/browse/IGNITE-14606 > > > > The concept is to filter IPs based on a pattern or a blocklist in > > IPFinders while consuming IPs. This is more pertinent for cloud based > > IPFinders since they can have shared containers. > > > > For the moment, I have implemented regex based filtering: > > > > https://issues.apache.org/jira/browse/IGNITE-14607 > > > > for Azure Blob Storage IP Finder. Over time, we can extend the same to > > other IP finders. > > > > Please see the PR: > > > > https://github.com/apache/ignite/pull/9024 > > > > Regards, > > > > Atri > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > >
IP Filtering in IPFinders
Hi All, I have opened the following JIRA for the said topic: https://issues.apache.org/jira/browse/IGNITE-14606 The concept is to filter IPs based on a pattern or a blocklist in IPFinders while consuming IPs. This is more pertinent for cloud based IPFinders since they can have shared containers. For the moment, I have implemented regex based filtering: https://issues.apache.org/jira/browse/IGNITE-14607 for Azure Blob Storage IP Finder. Over time, we can extend the same to other IP finders. Please see the PR: https://github.com/apache/ignite/pull/9024 Regards, Atri -- Regards, Atri Apache Concerted
[jira] [Created] (IGNITE-14608) Blocklist Based IP Filtering in IPFinders
Atri Sharma created IGNITE-14608: Summary: Blocklist Based IP Filtering in IPFinders Key: IGNITE-14608 URL: https://issues.apache.org/jira/browse/IGNITE-14608 Project: Ignite Issue Type: Sub-task Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14607) Regex Based Filter in IPFinders
Atri Sharma created IGNITE-14607: Summary: Regex Based Filter in IPFinders Key: IGNITE-14607 URL: https://issues.apache.org/jira/browse/IGNITE-14607 Project: Ignite Issue Type: Sub-task Reporter: Atri Sharma This Jira tracks the effort to implement regex based IP filtering in IPFinders -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14606) Filter Support in IPFinders
Atri Sharma created IGNITE-14606: Summary: Filter Support in IPFinders Key: IGNITE-14606 URL: https://issues.apache.org/jira/browse/IGNITE-14606 Project: Ignite Issue Type: Improvement Reporter: Atri Sharma In some scenarios, IPFinders need to be able to filter returned IPs based on a blocklist or a pattern. This is especially useful for cloud based IPFinders where the container can be shared. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Azure Cloud IP Finder
Thank you so much! Special thanks to yourself for suggesting this, Ilya for reviewing and helping close it out and Sam for helping me when I was stuck :) On Fri, 16 Apr 2021, 22:38 Denis Magda, wrote: > Congrats, Atri! > > Bragged about it a bit: > https://twitter.com/denismagda/status/1383104415087927297?s=20 > > -- > Denis Magda, > VP, Developer Relations and Product Marketing > > <https://ignite-summit.org> > > > On Fri, Apr 16, 2021 at 12:57 PM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > wrote: > > > Hello! > > > > Thank you for driving this effort, Atri. > > > > The Azure IP Finder code has been merged and will surely be a highlight > in > > Apache Ignite 2.11. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > ср, 14 апр. 2021 г. в 11:54, Atri Sharma : > > > > > Thanks for your comments. > > > > > > I have moved the dependency versions for shared versions to parent POM > > > and also for the majorly used versions for module specific > > > dependencies. > > > > > > I have also fixed the Ipv6 issue and have tested with both Ipv4 and > > > Ipv6 and it works fine. > > > > > > Please let me know your thoughts. > > > > > > Atri > > > > > > On Tue, Apr 13, 2021 at 9:39 PM Ilya Kasnacheev > > > wrote: > > > > > > > > Hello! > > > > > > > > I have added more comments to the PR. > > > > > > > > IPv6 still needs to be supported and dependencies' versions should be > > > moved > > > > to parent/pom.xml, at least for shared dependencies such as log4j and > > > > jackson, and preferably for all dependencies. > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > вт, 13 апр. 2021 г. в 10:33, Atri Sharma : > > > > > > > > > Hello, > > > > > > > > > > Sorry for the late reply. > > > > > > > > > > Thank you for taking a look. Indeed, there were some issues and > they > > > > > have now been fixed. > > > > > > > > > > I am able to start a two node cluster with Azure IPFinder enabled > and > > > > > shut it down successfully (using Ipv4). > > > > > > > > > > Please see the latest iteration and let me know your thoughts and > > > comments. > > > > > > > > > > Regards, > > > > > > > > > > Atri > > > > > > > > > > On Fri, Apr 9, 2021 at 4:58 PM Ilya Kasnacheev > > > > > wrote: > > > > > > > > > > > > Hello! > > > > > > > > > > > > I have responded to the ticket after testing on live Azure. > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > пт, 9 апр. 2021 г. в 08:37, Atri Sharma : > > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > > > > > Please let me know if I can help with any further iterations on > > > the PR. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Atri > > > > > > > > > > > > > > On Wed, Apr 7, 2021 at 5:04 PM Atri Sharma > > > wrote: > > > > > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > > > > > > > Thanks for taking a look. I was able to resolve dependencies > > > (Thanks, > > > > > > > > Sam!) and have updated the PR. > > > > > > > > > > > > > > > > Copying the jars from ignite-azure to libs works for me. > > > > > > > > > > > > > > > > Please see and let me know your thoughts. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Atri > > > > > > > > > > > > > > > > On Mon, Apr 5, 2021 at 9:24 PM Ilya Kasnacheev > > > > > > > > wrote: > > > > > > > > > > > > >
Re: Stop sending IGNITE Created e-mails to dev@
+1 to move issues to the issues list. For MTCGA, maybe build@? On Wed, Apr 14, 2021 at 8:35 PM Ilya Kasnacheev wrote: > > Hello! > > We have a discussion on how to ensure best engagement in dev@ list, and it > seems that Issue Created emails from IGNITE project consume a lot of screen > space, it's hard to spot genuine discussions in > https://lists.apache.org/list.html?dev@ignite.apache.org for example. > > We already have issues@ mailing list. I propose that we stop sending any > JIRA emails to dev@. If anyone wishes to get just Created emails, they can > subscribe to these messages in their JIRA account settings. I imagine most > of you already filter these messages out, so you may need to adjust your > filters slightly. > > A distant second is MTCGA messages, which are also autogenerated and not > informative for most readers of the channel, since they are at best > targeted at a single committer and at worst flaky. > > Where could we move those? What is your opinion here, on both issues? > > Regards, -- Regards, Atri Apache Concerted
Re: Azure Cloud IP Finder
Thanks for your comments. I have moved the dependency versions for shared versions to parent POM and also for the majorly used versions for module specific dependencies. I have also fixed the Ipv6 issue and have tested with both Ipv4 and Ipv6 and it works fine. Please let me know your thoughts. Atri On Tue, Apr 13, 2021 at 9:39 PM Ilya Kasnacheev wrote: > > Hello! > > I have added more comments to the PR. > > IPv6 still needs to be supported and dependencies' versions should be moved > to parent/pom.xml, at least for shared dependencies such as log4j and > jackson, and preferably for all dependencies. > > Regards, > -- > Ilya Kasnacheev > > > вт, 13 апр. 2021 г. в 10:33, Atri Sharma : > > > Hello, > > > > Sorry for the late reply. > > > > Thank you for taking a look. Indeed, there were some issues and they > > have now been fixed. > > > > I am able to start a two node cluster with Azure IPFinder enabled and > > shut it down successfully (using Ipv4). > > > > Please see the latest iteration and let me know your thoughts and comments. > > > > Regards, > > > > Atri > > > > On Fri, Apr 9, 2021 at 4:58 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > I have responded to the ticket after testing on live Azure. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > пт, 9 апр. 2021 г. в 08:37, Atri Sharma : > > > > > > > Hi Ilya, > > > > > > > > Please let me know if I can help with any further iterations on the PR. > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > On Wed, Apr 7, 2021 at 5:04 PM Atri Sharma wrote: > > > > > > > > > > Hi Ilya, > > > > > > > > > > Thanks for taking a look. I was able to resolve dependencies (Thanks, > > > > > Sam!) and have updated the PR. > > > > > > > > > > Copying the jars from ignite-azure to libs works for me. > > > > > > > > > > Please see and let me know your thoughts. > > > > > > > > > > Regards, > > > > > > > > > > Atri > > > > > > > > > > On Mon, Apr 5, 2021 at 9:24 PM Ilya Kasnacheev > > > > > wrote: > > > > > > > > > > > > Hello again! > > > > > > > > > > > > I re-checked our cloud discovery options by moving ignite-aws, > > > > ignite-gce, > > > > > > ignite-cloud directories from lib/optional to lib/ and trying to > > run > > > > Ignite > > > > > > with simple config taken from examples. > > > > > > > > > > > > The results are the following: > > > > > > ignite-aws seems to work (complains about unknown key) > > > > > > ignite-gce seems to work too (complains about zero length key, > > this is > > > > > > before it tries any network access so maybe there are other issues > > > > down the > > > > > > path) > > > > > > ignite-cloud (jclouds) DOES NOT work. Filed > > > > > > https://issues.apache.org/jira/browse/IGNITE-14481 . I guess this > > > > discovery > > > > > > finder is not widely used. > > > > > > ignite-azure, presently, will not work too. Please use ignite-aws > > as an > > > > > > example since it sees the most usage. > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > пн, 5 апр. 2021 г. в 16:05, Ilya Kasnacheev < > > ilya.kasnach...@gmail.com > > > > >: > > > > > > > > > > > > > Hello! > > > > > > > > > > > > > > I'm not sure that I can see any attachment to your e-mail. Can > > you > > > > please > > > > > > > re-send? > > > > > > > > > > > > > > We could have broken some of those, I guess, since we seem to > > not run > > > > > > > integration tests for them. > > > > > > > > > > > > > > Regards, > > > > > > > -- > > > > > > > Ilya Kasnacheev > >
Re: Azure Cloud IP Finder
Hello, Sorry for the late reply. Thank you for taking a look. Indeed, there were some issues and they have now been fixed. I am able to start a two node cluster with Azure IPFinder enabled and shut it down successfully (using Ipv4). Please see the latest iteration and let me know your thoughts and comments. Regards, Atri On Fri, Apr 9, 2021 at 4:58 PM Ilya Kasnacheev wrote: > > Hello! > > I have responded to the ticket after testing on live Azure. > > Regards, > -- > Ilya Kasnacheev > > > пт, 9 апр. 2021 г. в 08:37, Atri Sharma : > > > Hi Ilya, > > > > Please let me know if I can help with any further iterations on the PR. > > > > Regards, > > > > Atri > > > > On Wed, Apr 7, 2021 at 5:04 PM Atri Sharma wrote: > > > > > > Hi Ilya, > > > > > > Thanks for taking a look. I was able to resolve dependencies (Thanks, > > > Sam!) and have updated the PR. > > > > > > Copying the jars from ignite-azure to libs works for me. > > > > > > Please see and let me know your thoughts. > > > > > > Regards, > > > > > > Atri > > > > > > On Mon, Apr 5, 2021 at 9:24 PM Ilya Kasnacheev > > > wrote: > > > > > > > > Hello again! > > > > > > > > I re-checked our cloud discovery options by moving ignite-aws, > > ignite-gce, > > > > ignite-cloud directories from lib/optional to lib/ and trying to run > > Ignite > > > > with simple config taken from examples. > > > > > > > > The results are the following: > > > > ignite-aws seems to work (complains about unknown key) > > > > ignite-gce seems to work too (complains about zero length key, this is > > > > before it tries any network access so maybe there are other issues > > down the > > > > path) > > > > ignite-cloud (jclouds) DOES NOT work. Filed > > > > https://issues.apache.org/jira/browse/IGNITE-14481 . I guess this > > discovery > > > > finder is not widely used. > > > > ignite-azure, presently, will not work too. Please use ignite-aws as an > > > > example since it sees the most usage. > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > пн, 5 апр. 2021 г. в 16:05, Ilya Kasnacheev > >: > > > > > > > > > Hello! > > > > > > > > > > I'm not sure that I can see any attachment to your e-mail. Can you > > please > > > > > re-send? > > > > > > > > > > We could have broken some of those, I guess, since we seem to not run > > > > > integration tests for them. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > пт, 2 апр. 2021 г. в 12:59, Atri Sharma : > > > > > > > > > >> Hello, > > > > >> > > > > >> Thank you for sharing. > > > > >> > > > > >> I was finally able to replicate the issue. However, I tried with > > other > > > > >> IPFinders and ran into the same problem (attached are the logs). > > > > >> > > > > >> Not sure what is causing this? > > > > >> > > > > >> Atri > > > > >> > > > > >> On Fri, Apr 2, 2021 at 2:29 PM Ilya Kasnacheev > > > > >> wrote: > > > > >> > > > > > >> > Hello! > > > > >> > > > > > >> > Please find attached the log file of errors. This is yesterday's > > (Apr > > > > >> 1) build. > > > > >> > > > > > >> > Regards, > > > > >> > -- > > > > >> > Ilya Kasnacheev > > > > >> > > > > > >> > > > > > >> > пт, 2 апр. 2021 г. в 11:52, Atri Sharma : > > > > >> >> > > > > >> >> I was able to, but then, it might be that something is cached > > locally. > > > > >> >> > > > > >> >> What errors did you run into? (I am assuming you are trying with > > a > > > > >> >> build from the latest iteration of the PR). > > > > >> >> > > > > >> >
Re: Azure Cloud IP Finder
Hi Ilya, Please let me know if I can help with any further iterations on the PR. Regards, Atri On Wed, Apr 7, 2021 at 5:04 PM Atri Sharma wrote: > > Hi Ilya, > > Thanks for taking a look. I was able to resolve dependencies (Thanks, > Sam!) and have updated the PR. > > Copying the jars from ignite-azure to libs works for me. > > Please see and let me know your thoughts. > > Regards, > > Atri > > On Mon, Apr 5, 2021 at 9:24 PM Ilya Kasnacheev > wrote: > > > > Hello again! > > > > I re-checked our cloud discovery options by moving ignite-aws, ignite-gce, > > ignite-cloud directories from lib/optional to lib/ and trying to run Ignite > > with simple config taken from examples. > > > > The results are the following: > > ignite-aws seems to work (complains about unknown key) > > ignite-gce seems to work too (complains about zero length key, this is > > before it tries any network access so maybe there are other issues down the > > path) > > ignite-cloud (jclouds) DOES NOT work. Filed > > https://issues.apache.org/jira/browse/IGNITE-14481 . I guess this discovery > > finder is not widely used. > > ignite-azure, presently, will not work too. Please use ignite-aws as an > > example since it sees the most usage. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пн, 5 апр. 2021 г. в 16:05, Ilya Kasnacheev : > > > > > Hello! > > > > > > I'm not sure that I can see any attachment to your e-mail. Can you please > > > re-send? > > > > > > We could have broken some of those, I guess, since we seem to not run > > > integration tests for them. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > пт, 2 апр. 2021 г. в 12:59, Atri Sharma : > > > > > >> Hello, > > >> > > >> Thank you for sharing. > > >> > > >> I was finally able to replicate the issue. However, I tried with other > > >> IPFinders and ran into the same problem (attached are the logs). > > >> > > >> Not sure what is causing this? > > >> > > >> Atri > > >> > > >> On Fri, Apr 2, 2021 at 2:29 PM Ilya Kasnacheev > > >> wrote: > > >> > > > >> > Hello! > > >> > > > >> > Please find attached the log file of errors. This is yesterday's (Apr > > >> 1) build. > > >> > > > >> > Regards, > > >> > -- > > >> > Ilya Kasnacheev > > >> > > > >> > > > >> > пт, 2 апр. 2021 г. в 11:52, Atri Sharma : > > >> >> > > >> >> I was able to, but then, it might be that something is cached locally. > > >> >> > > >> >> What errors did you run into? (I am assuming you are trying with a > > >> >> build from the latest iteration of the PR). > > >> >> > > >> >> Atri > > >> >> > > >> >> On Fri, Apr 2, 2021 at 2:19 PM Ilya Kasnacheev > > >> >> wrote: > > >> >> > > > >> >> > Hello! > > >> >> > > > >> >> > But are you successful in running a node with Azure IP finder > > >> enabled in > > >> >> > configuration, starting from that release directory? > > >> >> > > > >> >> > I amn't. > > >> >> > > > >> >> > Regards, > > >> >> > -- > > >> >> > Ilya Kasnacheev > > >> >> > > > >> >> > > > >> >> > пт, 2 апр. 2021 г. в 07:42, Atri Sharma : > > >> >> > > > >> >> > > Hello, > > >> >> > > > > >> >> > > Thank you for taking a look. > > >> >> > > > > >> >> > > I am not sure what the confusion is. On the current iteration of > > >> PR, I > > >> >> > > am able to build and release and looking in the location you > > >> >> > > mentioned, I see: > > >> >> > > > > >> >> > > Atris-MacBook-Pro-15:libs atrisharma$ cd optional/ > > >> >> > > > > >> >> > > Atris-MacBook-Pro-15:optional atrisharma$ cd ignite-azure/ > > >> >> > > > > >>
Re: Azure Cloud IP Finder
Hi Ilya, Thanks for taking a look. I was able to resolve dependencies (Thanks, Sam!) and have updated the PR. Copying the jars from ignite-azure to libs works for me. Please see and let me know your thoughts. Regards, Atri On Mon, Apr 5, 2021 at 9:24 PM Ilya Kasnacheev wrote: > > Hello again! > > I re-checked our cloud discovery options by moving ignite-aws, ignite-gce, > ignite-cloud directories from lib/optional to lib/ and trying to run Ignite > with simple config taken from examples. > > The results are the following: > ignite-aws seems to work (complains about unknown key) > ignite-gce seems to work too (complains about zero length key, this is > before it tries any network access so maybe there are other issues down the > path) > ignite-cloud (jclouds) DOES NOT work. Filed > https://issues.apache.org/jira/browse/IGNITE-14481 . I guess this discovery > finder is not widely used. > ignite-azure, presently, will not work too. Please use ignite-aws as an > example since it sees the most usage. > > Regards, > -- > Ilya Kasnacheev > > > пн, 5 апр. 2021 г. в 16:05, Ilya Kasnacheev : > > > Hello! > > > > I'm not sure that I can see any attachment to your e-mail. Can you please > > re-send? > > > > We could have broken some of those, I guess, since we seem to not run > > integration tests for them. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пт, 2 апр. 2021 г. в 12:59, Atri Sharma : > > > >> Hello, > >> > >> Thank you for sharing. > >> > >> I was finally able to replicate the issue. However, I tried with other > >> IPFinders and ran into the same problem (attached are the logs). > >> > >> Not sure what is causing this? > >> > >> Atri > >> > >> On Fri, Apr 2, 2021 at 2:29 PM Ilya Kasnacheev > >> wrote: > >> > > >> > Hello! > >> > > >> > Please find attached the log file of errors. This is yesterday's (Apr > >> 1) build. > >> > > >> > Regards, > >> > -- > >> > Ilya Kasnacheev > >> > > >> > > >> > пт, 2 апр. 2021 г. в 11:52, Atri Sharma : > >> >> > >> >> I was able to, but then, it might be that something is cached locally. > >> >> > >> >> What errors did you run into? (I am assuming you are trying with a > >> >> build from the latest iteration of the PR). > >> >> > >> >> Atri > >> >> > >> >> On Fri, Apr 2, 2021 at 2:19 PM Ilya Kasnacheev > >> >> wrote: > >> >> > > >> >> > Hello! > >> >> > > >> >> > But are you successful in running a node with Azure IP finder > >> enabled in > >> >> > configuration, starting from that release directory? > >> >> > > >> >> > I amn't. > >> >> > > >> >> > Regards, > >> >> > -- > >> >> > Ilya Kasnacheev > >> >> > > >> >> > > >> >> > пт, 2 апр. 2021 г. в 07:42, Atri Sharma : > >> >> > > >> >> > > Hello, > >> >> > > > >> >> > > Thank you for taking a look. > >> >> > > > >> >> > > I am not sure what the confusion is. On the current iteration of > >> PR, I > >> >> > > am able to build and release and looking in the location you > >> >> > > mentioned, I see: > >> >> > > > >> >> > > Atris-MacBook-Pro-15:libs atrisharma$ cd optional/ > >> >> > > > >> >> > > Atris-MacBook-Pro-15:optional atrisharma$ cd ignite-azure/ > >> >> > > > >> >> > > Atris-MacBook-Pro-15:ignite-azure atrisharma$ ls > >> >> > > > >> >> > > README.txt azure-storage-blob-12.0.0.jar > >> ignite-azure-2.11.0-SNAPSHOT.jar > >> >> > > > >> >> > > > >> >> > > Attached is the screenshot. > >> >> > > > >> >> > > I have fixed the comments on the PR, please see and let me know. > >> >> > > > >> >> > > Atri > >> >> > > > >> >> > > On Thu, Apr 1, 2021 at 8:21 PM Ilya Kasnacheev > >> >> > > wrote: > >> >> > > > > &
Re: [DISCUSSION] Java thin client: Continuous Queries public API
I do agree with your proposition, but the idea of an interface being exposed through documentation seems a bit clunky to me. On Mon, Apr 5, 2021 at 12:40 PM Alex Plehanov wrote: > > Hi Atri, > > The new interface is only needed for thin client, it's not a good idea to > add such a method to thick client too. > > пн, 5 апр. 2021 г. в 09:48, Atri Sharma : > > > Hi Alex, > > > > Sorry for the late reply. > > > > Regarding the thick client, would it be a challenge to add a new method > > which takes takes interface as parameter? That will not break the back > > compatible > > > > On Fri, 2 Apr 2021, 14:32 Alex Plehanov, wrote: > > > > > Hello, Atri > > > > > > 1. ClientChannelDisconnectListener is thin client specific. > > > 2. Thick client API uses JCache interfaces, which cannot be modified. > > > 2. We can't modify thick client public API either, due to backward > > > compatibility. > > > > > > пт, 2 апр. 2021 г. в 11:51, Atri Sharma : > > > > > > > Personally, I would disagree with an interface implementation being > > > > dictated by the documentation rather than the method signature. > > > > > > > > How about having a generic interface (placeholder interface), have > > > > both the thick and thin clients expose methods using the interface as > > > > parameters, then let ClientChannelDisconnectListener actually > > > > implement that interface and override the base methods? The methods > > > > can be no-op in the thick clients. > > > > > > > > On Fri, Apr 2, 2021 at 2:04 PM Alex Plehanov > > > > wrote: > > > > > > > > > > Hello, Igniters! > > > > > > > > > > I'd like to discuss java thin client Continuous Queries public API. > > > > > > > > > > Currently, the thin client is not JCache compatible and without > > JCache > > > > > compatible cache instance we can't use classes and API which already > > > used > > > > > by the thick client for cache entry events listening. > > > > > In my first attempt to implement thin client CQ I've tried to create > > > own > > > > CQ > > > > > classes and interfaces for thin client, but such API looks weird. On > > > the > > > > > one hand, we should use CacheEntryEventFilter (JCache interface) > > since > > > > it's > > > > > required by the server-side, on the other hand, we can't use > > > > > CacheEntryUpdatedListener since it requires CacheEntryEvent which > > > > requires > > > > > an instance of Cache (JCache interface), which doesn't exist on the > > > > > thin-client side. > > > > > We've already discussed this problem with Pavel Tupitsyn in ticket > > [1] > > > > and > > > > > come to an agreement that the most suitable solution is to implement > > > some > > > > > private thin-client cache to JCache adapter, but not expose it to > > > public > > > > > API and don't provide full JCache support by the thin client (see > > > > comments > > > > > in the ticket for more details). In this case, we can reuse CQ > > classes > > > > and > > > > > interfaces and make the API similar to thick-client. > > > > > > > > > > Another problem: for the thin client we should be able to inform the > > > user > > > > > about channel failure. I propose to introduce some interface > > > > > ClientChannelDisconnectListener and notify about channel failure if > > > > > provided CacheEntryListener implements this interface. Unfortunately, > > > if > > > > we > > > > > want to keep thin client API similar to the thick client we can't > > > > > explicitly use this interface in methods parameters, so the knowledge > > > > that > > > > > user can use this interface for cache entry listener can be obtained > > > only > > > > > from JavaDoc or Ignite documentation. > > > > > > > > > > Igniters, WDYT? > > > > > > > > > > Here is PR with implemented proposals [2]. > > > > > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-14402 > > > > > [2]: https://github.com/apache/ignite/pull/8960 > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > -- Regards, Atri Apache Concerted
Re: [DISCUSSION] Java thin client: Continuous Queries public API
Hi Alex, Sorry for the late reply. Regarding the thick client, would it be a challenge to add a new method which takes takes interface as parameter? That will not break the back compatible On Fri, 2 Apr 2021, 14:32 Alex Plehanov, wrote: > Hello, Atri > > 1. ClientChannelDisconnectListener is thin client specific. > 2. Thick client API uses JCache interfaces, which cannot be modified. > 2. We can't modify thick client public API either, due to backward > compatibility. > > пт, 2 апр. 2021 г. в 11:51, Atri Sharma : > > > Personally, I would disagree with an interface implementation being > > dictated by the documentation rather than the method signature. > > > > How about having a generic interface (placeholder interface), have > > both the thick and thin clients expose methods using the interface as > > parameters, then let ClientChannelDisconnectListener actually > > implement that interface and override the base methods? The methods > > can be no-op in the thick clients. > > > > On Fri, Apr 2, 2021 at 2:04 PM Alex Plehanov > > wrote: > > > > > > Hello, Igniters! > > > > > > I'd like to discuss java thin client Continuous Queries public API. > > > > > > Currently, the thin client is not JCache compatible and without JCache > > > compatible cache instance we can't use classes and API which already > used > > > by the thick client for cache entry events listening. > > > In my first attempt to implement thin client CQ I've tried to create > own > > CQ > > > classes and interfaces for thin client, but such API looks weird. On > the > > > one hand, we should use CacheEntryEventFilter (JCache interface) since > > it's > > > required by the server-side, on the other hand, we can't use > > > CacheEntryUpdatedListener since it requires CacheEntryEvent which > > requires > > > an instance of Cache (JCache interface), which doesn't exist on the > > > thin-client side. > > > We've already discussed this problem with Pavel Tupitsyn in ticket [1] > > and > > > come to an agreement that the most suitable solution is to implement > some > > > private thin-client cache to JCache adapter, but not expose it to > public > > > API and don't provide full JCache support by the thin client (see > > comments > > > in the ticket for more details). In this case, we can reuse CQ classes > > and > > > interfaces and make the API similar to thick-client. > > > > > > Another problem: for the thin client we should be able to inform the > user > > > about channel failure. I propose to introduce some interface > > > ClientChannelDisconnectListener and notify about channel failure if > > > provided CacheEntryListener implements this interface. Unfortunately, > if > > we > > > want to keep thin client API similar to the thick client we can't > > > explicitly use this interface in methods parameters, so the knowledge > > that > > > user can use this interface for cache entry listener can be obtained > only > > > from JavaDoc or Ignite documentation. > > > > > > Igniters, WDYT? > > > > > > Here is PR with implemented proposals [2]. > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-14402 > > > [2]: https://github.com/apache/ignite/pull/8960 > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > >
Re: Azure Cloud IP Finder
Hello,
Thank you for sharing.
I was finally able to replicate the issue. However, I tried with other
IPFinders and ran into the same problem (attached are the logs).
Not sure what is causing this?
Atri
On Fri, Apr 2, 2021 at 2:29 PM Ilya Kasnacheev
wrote:
>
> Hello!
>
> Please find attached the log file of errors. This is yesterday's (Apr 1)
> build.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 2 апр. 2021 г. в 11:52, Atri Sharma :
>>
>> I was able to, but then, it might be that something is cached locally.
>>
>> What errors did you run into? (I am assuming you are trying with a
>> build from the latest iteration of the PR).
>>
>> Atri
>>
>> On Fri, Apr 2, 2021 at 2:19 PM Ilya Kasnacheev
>> wrote:
>> >
>> > Hello!
>> >
>> > But are you successful in running a node with Azure IP finder enabled in
>> > configuration, starting from that release directory?
>> >
>> > I amn't.
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пт, 2 апр. 2021 г. в 07:42, Atri Sharma :
>> >
>> > > Hello,
>> > >
>> > > Thank you for taking a look.
>> > >
>> > > I am not sure what the confusion is. On the current iteration of PR, I
>> > > am able to build and release and looking in the location you
>> > > mentioned, I see:
>> > >
>> > > Atris-MacBook-Pro-15:libs atrisharma$ cd optional/
>> > >
>> > > Atris-MacBook-Pro-15:optional atrisharma$ cd ignite-azure/
>> > >
>> > > Atris-MacBook-Pro-15:ignite-azure atrisharma$ ls
>> > >
>> > > README.txt azure-storage-blob-12.0.0.jar ignite-azure-2.11.0-SNAPSHOT.jar
>> > >
>> > >
>> > > Attached is the screenshot.
>> > >
>> > > I have fixed the comments on the PR, please see and let me know.
>> > >
>> > > Atri
>> > >
>> > > On Thu, Apr 1, 2021 at 8:21 PM Ilya Kasnacheev
>> > > wrote:
>> > > >
>> > > > Hello!
>> > > >
>> > > > Were you successful with using it from the deliverable of mvn
>> > > > initialize
>> > > > -Prelease?
>> > > >
>> > > > Please refer for example to aws module dependencies section:
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-core
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-s3
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-ec2
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-elasticloadbalancing
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-elasticloadbalancingv2
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-java-sdk-kms
>> > > > ${aws.sdk.version}
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > com.fasterxml.jackson.core
>> > > > jackson-core
>> > > > ${jackson.version}
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > com.fasterxml.jackson.core
>> > > > jackson-annotations
>> > > > ${jackson.version}
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > com.fasterxml.jackson.core
>> > > > jackson-databind
>> > > > ${jackson.version}
>> > > >
>> > > >
>> > > >
>> > > > com.amazonaws
>> > > > aws-encryption-sdk-java
>> > > > ${aws.encryption.sdk.version}
>> > > >
>> > > >
>> > > > org.bouncycastle
>> >
Re: Azure Cloud IP Finder
I was able to, but then, it might be that something is cached locally.
What errors did you run into? (I am assuming you are trying with a
build from the latest iteration of the PR).
Atri
On Fri, Apr 2, 2021 at 2:19 PM Ilya Kasnacheev
wrote:
>
> Hello!
>
> But are you successful in running a node with Azure IP finder enabled in
> configuration, starting from that release directory?
>
> I amn't.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 2 апр. 2021 г. в 07:42, Atri Sharma :
>
> > Hello,
> >
> > Thank you for taking a look.
> >
> > I am not sure what the confusion is. On the current iteration of PR, I
> > am able to build and release and looking in the location you
> > mentioned, I see:
> >
> > Atris-MacBook-Pro-15:libs atrisharma$ cd optional/
> >
> > Atris-MacBook-Pro-15:optional atrisharma$ cd ignite-azure/
> >
> > Atris-MacBook-Pro-15:ignite-azure atrisharma$ ls
> >
> > README.txt azure-storage-blob-12.0.0.jar ignite-azure-2.11.0-SNAPSHOT.jar
> >
> >
> > Attached is the screenshot.
> >
> > I have fixed the comments on the PR, please see and let me know.
> >
> > Atri
> >
> > On Thu, Apr 1, 2021 at 8:21 PM Ilya Kasnacheev
> > wrote:
> > >
> > > Hello!
> > >
> > > Were you successful with using it from the deliverable of mvn initialize
> > > -Prelease?
> > >
> > > Please refer for example to aws module dependencies section:
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-core
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-s3
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-ec2
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-elasticloadbalancing
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-elasticloadbalancingv2
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-java-sdk-kms
> > > ${aws.sdk.version}
> > >
> > >
> > >
> > >
> > > com.fasterxml.jackson.core
> > > jackson-core
> > > ${jackson.version}
> > >
> > >
> > >
> > >
> > > com.fasterxml.jackson.core
> > > jackson-annotations
> > > ${jackson.version}
> > >
> > >
> > >
> > >
> > > com.fasterxml.jackson.core
> > > jackson-databind
> > > ${jackson.version}
> > >
> > >
> > >
> > > com.amazonaws
> > > aws-encryption-sdk-java
> > > ${aws.encryption.sdk.version}
> > >
> > >
> > > org.bouncycastle
> > > bcprov-ext-jdk15on
> > >
> > >
> > >
> > >
> > >
> > > org.bouncycastle
> > > bcprov-ext-jdk15on
> > > ${bouncycastle.version}
> > >
> > >
> > >
> > > joda-time
> > > joda-time
> > > 2.8.1
> > >
> > >
> > >
> > > org.apache.httpcomponents
> > > httpclient
> > > ${httpclient.version}
> > >
> > >
> > >
> > > org.apache.httpcomponents
> > > httpcore
> > > ${httpcore.version}
> > >
> > >
> > > azure module should likewise list all of its actual dependencies. Right
> > now
> > > the module directory is empty so I'm not even going to test to run it in
> > > stand-alone mode:
> > > ~/Downloads/apache-ignite-2.11.0-SNAPSHOT-bin% ls
> > libs/optional/ignite-azure
> > > azure-storage-blob-12.0.0.jar ignite-azure-2.11.0-SNAPSHOT.jar
> > README.txt
> > >
> > > Please address this issue along with other things which I have also
> > > commented on the PR.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 31 мар. 2021 г. в 08:17, Atri Sharma :
> > >
> > > > Thank you for the review.
> > > >
> > > > I have updated the PR. Please see.
> > &
Re: [DISCUSSION] Java thin client: Continuous Queries public API
Personally, I would disagree with an interface implementation being dictated by the documentation rather than the method signature. How about having a generic interface (placeholder interface), have both the thick and thin clients expose methods using the interface as parameters, then let ClientChannelDisconnectListener actually implement that interface and override the base methods? The methods can be no-op in the thick clients. On Fri, Apr 2, 2021 at 2:04 PM Alex Plehanov wrote: > > Hello, Igniters! > > I'd like to discuss java thin client Continuous Queries public API. > > Currently, the thin client is not JCache compatible and without JCache > compatible cache instance we can't use classes and API which already used > by the thick client for cache entry events listening. > In my first attempt to implement thin client CQ I've tried to create own CQ > classes and interfaces for thin client, but such API looks weird. On the > one hand, we should use CacheEntryEventFilter (JCache interface) since it's > required by the server-side, on the other hand, we can't use > CacheEntryUpdatedListener since it requires CacheEntryEvent which requires > an instance of Cache (JCache interface), which doesn't exist on the > thin-client side. > We've already discussed this problem with Pavel Tupitsyn in ticket [1] and > come to an agreement that the most suitable solution is to implement some > private thin-client cache to JCache adapter, but not expose it to public > API and don't provide full JCache support by the thin client (see comments > in the ticket for more details). In this case, we can reuse CQ classes and > interfaces and make the API similar to thick-client. > > Another problem: for the thin client we should be able to inform the user > about channel failure. I propose to introduce some interface > ClientChannelDisconnectListener and notify about channel failure if > provided CacheEntryListener implements this interface. Unfortunately, if we > want to keep thin client API similar to the thick client we can't > explicitly use this interface in methods parameters, so the knowledge that > user can use this interface for cache entry listener can be obtained only > from JavaDoc or Ignite documentation. > > Igniters, WDYT? > > Here is PR with implemented proposals [2]. > > [1]: https://issues.apache.org/jira/browse/IGNITE-14402 > [2]: https://github.com/apache/ignite/pull/8960 -- Regards, Atri Apache Concerted
Re: Azure Cloud IP Finder
Hello,
Thank you for taking a look.
I am not sure what the confusion is. On the current iteration of PR, I
am able to build and release and looking in the location you
mentioned, I see:
Atris-MacBook-Pro-15:libs atrisharma$ cd optional/
Atris-MacBook-Pro-15:optional atrisharma$ cd ignite-azure/
Atris-MacBook-Pro-15:ignite-azure atrisharma$ ls
README.txt azure-storage-blob-12.0.0.jar ignite-azure-2.11.0-SNAPSHOT.jar
Attached is the screenshot.
I have fixed the comments on the PR, please see and let me know.
Atri
On Thu, Apr 1, 2021 at 8:21 PM Ilya Kasnacheev
wrote:
>
> Hello!
>
> Were you successful with using it from the deliverable of mvn initialize
> -Prelease?
>
> Please refer for example to aws module dependencies section:
>
>
> com.amazonaws
> aws-java-sdk-core
> ${aws.sdk.version}
>
>
>
> com.amazonaws
> aws-java-sdk-s3
> ${aws.sdk.version}
>
>
>
> com.amazonaws
> aws-java-sdk-ec2
> ${aws.sdk.version}
>
>
>
> com.amazonaws
> aws-java-sdk-elasticloadbalancing
> ${aws.sdk.version}
>
>
>
> com.amazonaws
> aws-java-sdk-elasticloadbalancingv2
> ${aws.sdk.version}
>
>
>
> com.amazonaws
> aws-java-sdk-kms
> ${aws.sdk.version}
>
>
>
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
>
>
> com.fasterxml.jackson.core
> jackson-annotations
> ${jackson.version}
>
>
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
>
>
> com.amazonaws
> aws-encryption-sdk-java
> ${aws.encryption.sdk.version}
>
>
> org.bouncycastle
> bcprov-ext-jdk15on
>
>
>
>
>
> org.bouncycastle
> bcprov-ext-jdk15on
> ${bouncycastle.version}
>
>
>
> joda-time
> joda-time
> 2.8.1
>
>
>
> org.apache.httpcomponents
> httpclient
> ${httpclient.version}
>
>
>
> org.apache.httpcomponents
> httpcore
> ${httpcore.version}
>
>
> azure module should likewise list all of its actual dependencies. Right now
> the module directory is empty so I'm not even going to test to run it in
> stand-alone mode:
> ~/Downloads/apache-ignite-2.11.0-SNAPSHOT-bin% ls libs/optional/ignite-azure
> azure-storage-blob-12.0.0.jar ignite-azure-2.11.0-SNAPSHOT.jar README.txt
>
> Please address this issue along with other things which I have also
> commented on the PR.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 31 мар. 2021 г. в 08:17, Atri Sharma :
>
> > Thank you for the review.
> >
> > I have updated the PR. Please see.
> >
> > On Mon, Mar 29, 2021 at 8:27 PM Ilya Kasnacheev
> > wrote:
> > >
> > > Hello!
> > >
> > > I have left some more comments after trying this change on a real Azure
> > > cluster.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вт, 23 мар. 2021 г. в 18:32, Atri Sharma :
> > >
> > > > Thank you!
> > > >
> > > > I have updated the PR. Please see and let me know.
> > > >
> > > > On Tue, Mar 23, 2021 at 4:27 PM Ilya Kasnacheev
> > > > wrote:
> > > > >
> > > > > Hello!
> > > > >
> > > > > I am going to check this change out when I have time, using my Azure
> > > > > account.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > вт, 23 мар. 2021 г. в 07:20, Atri Sharma :
> > > > >
> > > > > > Gentle reminder on this -- please help in reviewing this.
> > > > > >
> > > > > > On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma
> > wrote:
> > > > > > >
> > > > > > > Thanks Denis.
> > > > > > >
> > > > > > > I have raised a PR for the same:
> > > > > > >
> > > > > > > https://github.com/apache/ignite/pull/8897
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Atri
> > > > > > >
> > > > > > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda
> > > > wrote:
> > > > > > > >
> > >
Re: Azure Cloud IP Finder
Thank you for the review. I have updated the PR. Please see. On Mon, Mar 29, 2021 at 8:27 PM Ilya Kasnacheev wrote: > > Hello! > > I have left some more comments after trying this change on a real Azure > cluster. > > Regards, > -- > Ilya Kasnacheev > > > вт, 23 мар. 2021 г. в 18:32, Atri Sharma : > > > Thank you! > > > > I have updated the PR. Please see and let me know. > > > > On Tue, Mar 23, 2021 at 4:27 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > I am going to check this change out when I have time, using my Azure > > > account. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > вт, 23 мар. 2021 г. в 07:20, Atri Sharma : > > > > > > > Gentle reminder on this -- please help in reviewing this. > > > > > > > > On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma wrote: > > > > > > > > > > Thanks Denis. > > > > > > > > > > I have raised a PR for the same: > > > > > > > > > > https://github.com/apache/ignite/pull/8897 > > > > > > > > > > Regards, > > > > > > > > > > Atri > > > > > > > > > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda > > wrote: > > > > > > > > > > > > Atri, > > > > > > > > > > > > Let's discuss the subj together with the community. Ignite already > > > > supports > > > > > > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is > > > > still > > > > > > missing. I can confirm that the demand exists, and rather > > frequently, > > > > I see > > > > > > developers asking for an Azure-native IP finder for Ignite. > > > > > > > > > > > > Atri, could you please research how to implement the IP finder and > > > > suggest > > > > > > a solution in this discussion thread? See how it was done for AWS > > and > > > > GCE, > > > > > > we might go the same route or use a more contemporary and > > > > easy-to-configure > > > > > > approach for Azure. > > > > > > > > > > > > [1] > > > > > > > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > > > > > > [2] > > > > > > > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Atri > > > > > Apache Concerted > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
uture from being completed by mistake. > > > > >> >> I think it will NOT be enough to do that returing the future to > > the > > > > >> >> end-user, but from every critical module to the outside of the > > > > module, > > > > >> >> e.g. to plugins. The impact of disclosing ExchangeFuture, > > > > >> >> PartitionReleaseFuture to plugins may be serious. > > > > >> >> > > > > >> >> IgniteFuture extends Future, CompletionStage which > > > > >> >> implementation > > > > >> >> will just wrap CompletableFuture these issues will be resolved > in > > > > >> natural > > > > >> >> way. > > > > >> >> In addition we can force toCompletableFuture() method to > return a > > > > >> >> defensive > > > > >> >> copy(), that resolves the last concern. > > > > >> >> > > > > >> >> > > > > >> >> On Fri, Mar 26, 2021 at 11:38 AM Konstantin Orlov > > > > >> >> > > > > >> >> wrote: > > > > >> >> > > > > >> >> > CompletableFuture seems a better option to me. > > > > >> >> > > > > > >> >> > -- > > > > >> >> > Regards, > > > > >> >> > Konstantin Orlov > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > On 26 Mar 2021, at 11:07, Pavel Tupitsyn < > > ptupit...@apache.org > > > > > > > > >> >> > > wrote: > > > > >> >> > > > > > > >> >> > > On the one hand, I agree with Alexey. > > > > >> >> > > CompletableFuture has complete* methods which should not be > > > > >> available > > > > >> >> to > > > > >> >> > > the user code. > > > > >> >> > > This can be solved with a simple interface like we do in > Thin > > > > >> Client: > > > > >> >> > > IgniteClientFuture extends Future, CompletionStage > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > On the other hand: > > > > >> >> > > - CompletionStage has toCompletableFuture anyway (rather > > weird) > > > > >> >> > > - Other libraries use CompletableFuture and it seems to be > > fine > > > > >> >> > > - Using CompletableFuture is the simplest approach > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > So I lean towards CompletableFuture too. > > > > >> >> > > > > > > >> >> > > On Fri, Mar 26, 2021 at 10:46 AM Alexey Kukushkin < > > > > >> >> > kukushkinale...@gmail.com> > > > > >> >> > > wrote: > > > > >> >> > > > > > > >> >> > >> I do not like Java's CompletableFuture and prefer our own > > > Future > > > > >> >> > (revised > > > > >> >> > >> IgniteFuture). > > > > >> >> > >> > > > > >> >> > >> My understanding of the Future (or Promise) pattern in > > general > > > > is > > > > >> >> having > > > > >> >> > >> two separate APIs: > > > > >> >> > >> > > > > >> >> > >> 1. Server-side: create, set result, raise error, cancel > > from > > > > >> >> > >> server. > > > > >> >> > >> 2. Client-side: get result, handle error, cancel from > > client > > > > >> >> > >> > > > > >> >> > >> Java's CompletableFuture looks like both the client-side > and > > > > >> >> > >> server-side API. The "Completeable" prefix in the name is > > > > already > > > > >> >> > confusing > > > > &
Re: Significant Items to Tackle
On further look, this looks interesting and I am happy to dive into the spring framework. Please help me in getting started On Mon, 29 Mar 2021, 11:49 Atri Sharma, wrote: > Thanks Sam and Ivan. > > I am happy to look into that but it looks like most of the work is done > there? > > Also, is there something in core that I can pickup as well? > > > On Mon, 29 Mar 2021, 03:20 Ivan Pavlukhin, wrote: > >> Gmail treated last message as spam. Bumping. >> >> 2021-03-28 16:05 GMT+03:00, Данилов Семён : >> > Hello Atri! >> > >> > Would you like to contribute into Ignite Spring integrations? You can >> take a >> > look at this particular jira filter: >> > >> https://issues.apache.org/jira/browse/IGNITE-9524?jql=project%20%3D%20Ignite%20and%20summary%20~%20%22spring%22%20and%20status%20not%20in%20(Resolved%2C%20Closed) >> > . >> > Personally, I think one of the most valuable issues there is >> > https://issues.apache.org/jira/browse/IGNITE-9524. I have a lot of >> > experience with Spring, so you can reach out to me if you have any >> > questions. >> > >> > Cheers, Sam. >> > >> > 25.03.2021, 23:33, "Atri Sharma" : >> >> Hi Community, >> >> >> >> First off, I want to thank the community for being so welcoming and >> >> helpful. This is an awesome place to be in. >> >> >> >> Now that I have worked on some issues, I would like to get my hands >> >> deeper >> >> and with larger issues in the core. Also, some more intermediate >> tickets >> >> to >> >> tackle, my queue has gotten empty :) >> >> >> >> Please help and advice. >> >> >> >> Regards, >> >> >> >> Atri >> > >> >> >> -- >> >> Best regards, >> Ivan Pavlukhin >> >
Re: Significant Items to Tackle
Thanks Sam and Ivan. I am happy to look into that but it looks like most of the work is done there? Also, is there something in core that I can pickup as well? On Mon, 29 Mar 2021, 03:20 Ivan Pavlukhin, wrote: > Gmail treated last message as spam. Bumping. > > 2021-03-28 16:05 GMT+03:00, Данилов Семён : > > Hello Atri! > > > > Would you like to contribute into Ignite Spring integrations? You can > take a > > look at this particular jira filter: > > > https://issues.apache.org/jira/browse/IGNITE-9524?jql=project%20%3D%20Ignite%20and%20summary%20~%20%22spring%22%20and%20status%20not%20in%20(Resolved%2C%20Closed) > > . > > Personally, I think one of the most valuable issues there is > > https://issues.apache.org/jira/browse/IGNITE-9524. I have a lot of > > experience with Spring, so you can reach out to me if you have any > > questions. > > > > Cheers, Sam. > > > > 25.03.2021, 23:33, "Atri Sharma" : > >> Hi Community, > >> > >> First off, I want to thank the community for being so welcoming and > >> helpful. This is an awesome place to be in. > >> > >> Now that I have worked on some issues, I would like to get my hands > >> deeper > >> and with larger issues in the core. Also, some more intermediate tickets > >> to > >> tackle, my queue has gotten empty :) > >> > >> Please help and advice. > >> > >> Regards, > >> > >> Atri > > > > > -- > > Best regards, > Ivan Pavlukhin >
Significant Items to Tackle
Hi Community, First off, I want to thank the community for being so welcoming and helpful. This is an awesome place to be in. Now that I have worked on some issues, I would like to get my hands deeper and with larger issues in the core. Also, some more intermediate tickets to tackle, my queue has gotten empty :) Please help and advice. Regards, Atri
Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.
I would suggest using CompletableFuture -- I don't see a need for a custom
interface that is unique to us.
It also allows a lower barrier for new contributors for understanding
existing code
On Thu, 25 Mar 2021, 20:18 Andrey Mashenkov,
wrote:
> Hi Igniters,
>
> I'd like to start a discussion about replacing our custom IgniteFuture
> class with CompletableFuture - existed JDK class
> or rework it's implementation (like some other products done) to a
> composition of CompletionStage and Future interfaces.
> or maybe other option if you have any ideas. Do you?
>
> 1. The first approach pros and cons are
> + Well-known JDK class
> + Already implemented
> - It is a class, not an interface.
> - Expose some potentially harmful methods like "complete()".
>
> On the other side, it has copy() method to create defensive copy and
> minimalCompletionStage() to restrict harmful method usage.
> Thus, this look like an applicable solution, but we should be careful
> exposing internal future to the outside.
>
> 2. The second approach is to implement our own interface like the next one:
>
> interface IgniteFuture extends CompletableStage, Future {
> }
>
> Pros and cons are
> + Our interfaces/classes contracts will expose an interface rather than
> concrete implementation.
> + All methods are safe.
> - Some implementation is required.
> - CompletableStage has a method toCompletableFuture() and can be converted
> to CompletableFuture. This should be supported.
>
> However, we still could wrap CompletableFuture and don't bother about
> creating a defensive copy.
>
>
> Other project experience:
> * Spotify uses CompletableFuture directly [1].
> * Redis goes the second approach [2]
> * Vertx explicitly extends CompletableFuture [3]. However, they have custom
> future classes and a number of helpers that could be replaced with
> CompletableStage. Maybe it is just a legacy.'
>
> Any thoughts?
>
> [1]
>
> https://spotify.github.io/completable-futures/apidocs/com/spotify/futures/ConcurrencyReducer.html
> [2]
>
> https://lettuce.io/lettuce-4/release/api/com/lambdaworks/redis/RedisFuture.html
> [3]
>
> https://javadoc.io/static/org.jspare.vertx/vertx-jspare/1.1.0-M03/org/jspare/vertx/concurrent/VertxCompletableFuture.html
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: Azure Cloud IP Finder
Thank you! I have updated the PR. Please see and let me know. On Tue, Mar 23, 2021 at 4:27 PM Ilya Kasnacheev wrote: > > Hello! > > I am going to check this change out when I have time, using my Azure > account. > > Regards, > -- > Ilya Kasnacheev > > > вт, 23 мар. 2021 г. в 07:20, Atri Sharma : > > > Gentle reminder on this -- please help in reviewing this. > > > > On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma wrote: > > > > > > Thanks Denis. > > > > > > I have raised a PR for the same: > > > > > > https://github.com/apache/ignite/pull/8897 > > > > > > Regards, > > > > > > Atri > > > > > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda wrote: > > > > > > > > Atri, > > > > > > > > Let's discuss the subj together with the community. Ignite already > > supports > > > > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is > > still > > > > missing. I can confirm that the demand exists, and rather frequently, > > I see > > > > developers asking for an Azure-native IP finder for Ignite. > > > > > > > > Atri, could you please research how to implement the IP finder and > > suggest > > > > a solution in this discussion thread? See how it was done for AWS and > > GCE, > > > > we might go the same route or use a more contemporary and > > easy-to-configure > > > > approach for Azure. > > > > > > > > [1] > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > > > > [2] > > > > > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > > > > > > > - > > > > Denis > > > > > > > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: Azure Cloud IP Finder
Gentle reminder on this -- please help in reviewing this. On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma wrote: > > Thanks Denis. > > I have raised a PR for the same: > > https://github.com/apache/ignite/pull/8897 > > Regards, > > Atri > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda wrote: > > > > Atri, > > > > Let's discuss the subj together with the community. Ignite already supports > > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is still > > missing. I can confirm that the demand exists, and rather frequently, I see > > developers asking for an Azure-native IP finder for Ignite. > > > > Atri, could you please research how to implement the IP finder and suggest > > a solution in this discussion thread? See how it was done for AWS and GCE, > > we might go the same route or use a more contemporary and easy-to-configure > > approach for Azure. > > > > [1] > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > > [2] > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > > > - > > Denis > > > > -- > Regards, > > Atri > Apache Concerted -- Regards, Atri Apache Concerted
Re: Azure Cloud IP Finder
Explanation of the design: The design follows a similar model to existing IP finders -- it either takes a pre-created container or creates a new container and creates entries in it for each IP address. I have tested it with a test account and the operation is fine. On Fri, Mar 19, 2021 at 10:23 AM Atri Sharma wrote: > > Thanks Denis. > > I have raised a PR for the same: > > https://github.com/apache/ignite/pull/8897 > > Regards, > > Atri > > On Wed, Mar 10, 2021 at 1:21 AM Denis Magda wrote: > > > > Atri, > > > > Let's discuss the subj together with the community. Ignite already supports > > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is still > > missing. I can confirm that the demand exists, and rather frequently, I see > > developers asking for an Azure-native IP finder for Ignite. > > > > Atri, could you please research how to implement the IP finder and suggest > > a solution in this discussion thread? See how it was done for AWS and GCE, > > we might go the same route or use a more contemporary and easy-to-configure > > approach for Azure. > > > > [1] > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > > [2] > > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > > > - > > Denis > > > > -- > Regards, > > Atri > Apache Concerted -- Regards, Atri Apache Concerted
[jira] [Created] (IGNITE-14347) Fix Node failure on Receiving Data of Unknown class via Distributed Metastorage
Atri Sharma created IGNITE-14347: Summary: Fix Node failure on Receiving Data of Unknown class via Distributed Metastorage Key: IGNITE-14347 URL: https://issues.apache.org/jira/browse/IGNITE-14347 Project: Ignite Issue Type: Improvement Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Azure Cloud IP Finder
Thanks Denis. I have raised a PR for the same: https://github.com/apache/ignite/pull/8897 Regards, Atri On Wed, Mar 10, 2021 at 1:21 AM Denis Magda wrote: > > Atri, > > Let's discuss the subj together with the community. Ignite already supports > AWS [1] and GCE [2] IP Finders out of the box, but the Azure one is still > missing. I can confirm that the demand exists, and rather frequently, I see > developers asking for an Azure-native IP finder for Ignite. > > Atri, could you please research how to implement the IP finder and suggest > a solution in this discussion thread? See how it was done for AWS and GCE, > we might go the same route or use a more contemporary and easy-to-configure > approach for Azure. > > [1] > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#amazon-s3-ip-finder > [2] > https://ignite.apache.org/docs/latest/clustering/discovery-in-the-cloud#google-compute-discovery > > - > Denis -- Regards, Atri Apache Concerted
[jira] [Created] (IGNITE-14346) Implement Azure Blob Storage Based IP Finder
Atri Sharma created IGNITE-14346: Summary: Implement Azure Blob Storage Based IP Finder Key: IGNITE-14346 URL: https://issues.apache.org/jira/browse/IGNITE-14346 Project: Ignite Issue Type: Improvement Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Thank you so much! On Thu, Mar 18, 2021 at 3:01 AM Valentin Kulichenko wrote: > > Hi Atri, > > Looks good to me, I've merged the changes. Please resolve the ticket. > > -Val > > On Wed, Mar 17, 2021 at 5:07 AM Atri Sharma wrote: > > > Hi Valentine, > > > > Hoping you are well. > > > > Please let me know if the PR looks ok. > > > > Regards, > > > > Atri > > > > > > On Thu, 11 Mar 2021, 12:09 Atri Sharma, wrote: > > > > > Hi Val, > > > > > > Thanks for taking a look. I have updated the PR, please see and let me > > > know your thoughts and feedback. > > > > > > Regards, > > > > > > Atri > > > > > > On Thu, Mar 11, 2021 at 6:16 AM Valentin Kulichenko > > > wrote: > > > > > > > > Atri, > > > > > > > > I've added my comments to the PR. > > > > > > > > -Val > > > > > > > > On Wed, Mar 10, 2021 at 2:44 AM Atri Sharma wrote: > > > > > > > > > I have just updated the PR: > > > > > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > > > > > Please review. > > > > > > > > > > On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma wrote: > > > > > > > > > > > > Hi Val, > > > > > > > > > > > > Thank you for taking the time on this one. > > > > > > > > > > > > The main reason as to why I chose that signature was because I felt > > > it > > > > > > gave a clear idea of the interface that the user should expect when > > > > > > using the method. So essentially, the user is providing a callback > > > > > > listener himself/herself and the API is just using that to tie the > > > > > > lifecycle of holding the semaphore. > > > > > > > > > > > > However, I do see your point and feel that the current signature > > that > > > > > > the PR has will limit the usability of the method and make users > > jump > > > > > > through more hoops. I have changed the signature to accept Callable > > > > > > returning T. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > > > > > > > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko > > > > > > wrote: > > > > > > > > > > > > > > Hi Atri, > > > > > > > > > > > > > > First and foremost, we need to clarify the API for this > > > functionality. > > > > > > > There are two options presented in the ticket, but no clear > > > decision > > > > > about > > > > > > > which one should be used. I personally think that the original > > > > > signature > > > > > > > (the one in the ticket description) makes more sense, but it > > looks > > > like > > > > > > > you've implemented an alternative suggested in the comments. What > > > was > > > > > your > > > > > > > motivation behind that? > > > > > > > > > > > > > > As for where the method can be located, I'm OK if we add it > > > directly > > > > > to the > > > > > > > IgniteSemaphore interface. > > > > > > > > > > > > > > -Val > > > > > > > > > > > > > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma > > > wrote: > > > > > > > > > > > > > > > Gentle ping > > > > > > > > > > > > > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > I would like to start a discussion around IGNITE-2399. > > > > > > > > > > > > > > > > > > Background: The typical use of IgniteSemaphore consists of > > > > > acquiring > > > > > > > > > the semaphore, performing the task and releasing the > > semaphore. > > > > > This > > > > > > > > > JIRA proposes a new method which will accept a callable, > > > acquire > > > > > the > > > > > > > > > semaphore, and return a future. Upon completion of the > > future, > > > the > > > > > > > > > semaphore is released. > > > > > > > > > > > > > > > > > > This API seems useful for an easy encapsulation of the > > > described > > > > > use > > > > > > > > > case and does not cause an internal flux since all the > > changes > > > are > > > > > > > > > focused at the public API. > > > > > > > > > > > > > > > > > > WIP PR for the same: > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > > > > > > > > > > > > > Please share your thoughts and comments. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Atri > > > > > > > > > Apache Concerted > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > Apache Concerted > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Atri > > > > > Apache Concerted > > > > > > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > -- Regards, Atri Apache Concerted
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Hi Valentine, Hoping you are well. Please let me know if the PR looks ok. Regards, Atri On Thu, 11 Mar 2021, 12:09 Atri Sharma, wrote: > Hi Val, > > Thanks for taking a look. I have updated the PR, please see and let me > know your thoughts and feedback. > > Regards, > > Atri > > On Thu, Mar 11, 2021 at 6:16 AM Valentin Kulichenko > wrote: > > > > Atri, > > > > I've added my comments to the PR. > > > > -Val > > > > On Wed, Mar 10, 2021 at 2:44 AM Atri Sharma wrote: > > > > > I have just updated the PR: > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > Please review. > > > > > > On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma wrote: > > > > > > > > Hi Val, > > > > > > > > Thank you for taking the time on this one. > > > > > > > > The main reason as to why I chose that signature was because I felt > it > > > > gave a clear idea of the interface that the user should expect when > > > > using the method. So essentially, the user is providing a callback > > > > listener himself/herself and the API is just using that to tie the > > > > lifecycle of holding the semaphore. > > > > > > > > However, I do see your point and feel that the current signature that > > > > the PR has will limit the usability of the method and make users jump > > > > through more hoops. I have changed the signature to accept Callable > > > > returning T. > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko > > > > wrote: > > > > > > > > > > Hi Atri, > > > > > > > > > > First and foremost, we need to clarify the API for this > functionality. > > > > > There are two options presented in the ticket, but no clear > decision > > > about > > > > > which one should be used. I personally think that the original > > > signature > > > > > (the one in the ticket description) makes more sense, but it looks > like > > > > > you've implemented an alternative suggested in the comments. What > was > > > your > > > > > motivation behind that? > > > > > > > > > > As for where the method can be located, I'm OK if we add it > directly > > > to the > > > > > IgniteSemaphore interface. > > > > > > > > > > -Val > > > > > > > > > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma > wrote: > > > > > > > > > > > Gentle ping > > > > > > > > > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > I would like to start a discussion around IGNITE-2399. > > > > > > > > > > > > > > Background: The typical use of IgniteSemaphore consists of > > > acquiring > > > > > > > the semaphore, performing the task and releasing the semaphore. > > > This > > > > > > > JIRA proposes a new method which will accept a callable, > acquire > > > the > > > > > > > semaphore, and return a future. Upon completion of the future, > the > > > > > > > semaphore is released. > > > > > > > > > > > > > > This API seems useful for an easy encapsulation of the > described > > > use > > > > > > > case and does not cause an internal flux since all the changes > are > > > > > > > focused at the public API. > > > > > > > > > > > > > > WIP PR for the same: > > > > > > > > > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > > > > > > > > > Please share your thoughts and comments. > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > > > > > > > > Atri > > > > > > > Apache Concerted > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > -- > Regards, > > Atri > Apache Concerted >
IGNITE-14314: GridDhtLockFuture#onComplete Should Be Aware of Cache Cleanup
Hi All, I would like to propose a change: today, in GridDhtLockFuture, we do not consider if the cache context is being cleaned up before loading missing entries from the store. However, if the cache context is being cleaned up, this operation is moot. I have raised a PR for the same. Please share your thoughts: https://github.com/apache/ignite/pull/8877 Regards, Atri -- Regards, Atri Apache Concerted
[jira] [Created] (IGNITE-14314) GridDhtLockFuture#onComplete Should Be Aware of Cache Cleanup
Atri Sharma created IGNITE-14314: Summary: GridDhtLockFuture#onComplete Should Be Aware of Cache Cleanup Key: IGNITE-14314 URL: https://issues.apache.org/jira/browse/IGNITE-14314 Project: Ignite Issue Type: Improvement Reporter: Atri Sharma Said method only takes node stoppage into account while deciding if store should be accessed. This can lead to race conditions when the cache is being cleaned up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request: IGNITE-8635: Add a Method to Inspect BinaryObject Size
Gentle ping On Thu, 11 Mar 2021 at 2:56 PM, Atri Sharma wrote: > Hi All, > > I have raised a PR for the above mentioned issue. Please see and help > review: > > https://github.com/apache/ignite/pull/8868 > > Regards, > > Atri > > -- > Regards, > > Atri > Apache Concerted > -- Regards, Atri Apache Concerted
Re: Re[2]: [VOTE] Release Apache Ignite 2.10.0 RC2
+1 (non binding)
Built from source, ran tests and tried the Java client
On Thu, 11 Mar 2021, 16:13 Zhenya Stanilovsky,
wrote:
>
>
> Build from sources, run .net tests, looks good. +1
>
>
>
> >+1 (binding)
> >
> >Downloaded binary packages, started nodes, .NET examples, .NET nodes.
> >Downloaded source package, built Java and .NET parts.
> >
> >On Thu, Mar 11, 2021 at 4:24 AM Maxim Muzafarov < mmu...@apache.org >
> wrote:
> >
> >> Dear Community,
> >>
> >> I have uploaded a release candidate to:
> >>
> >> https://dist.apache.org/repos/dist/dev/ignite/2.10.0-rc2/
> >> https://dist.apache.org/repos/dist/dev/ignite/packages_2.10.0-rc2/
> >>
> >> The following staging can be used for testing:
> >>
> https://repository.apache.org/content/repositories/orgapacheignite-1507/
> >>
> >>
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
> >>
> >> Tag name is 2.10.0-rc2:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.10.0-rc2
> >>
> >> RELEASE_NOTES:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.10
> >>
> >> Complete list of resolved issues:
> >>
> >> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20
> 'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.10'))
> >>
> >> DEVNOTES:
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.10
> >>
> >>
> >> Additional checks have been performed (available for users included into
> >> the release group on TeamCity).
> >>
> >> TC [Check RC: Licenses, compile, chksum]
> >>
> >>
> https://ci.ignite.apache.org/viewLog.html?buildId=5909007=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum
> >>
> >> TC [3] Build & Upload Nuget Staging Packages
> >>
> >>
> https://ci.ignite.apache.org/viewLog.html?buildId=5909005=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=artifacts
> >>
> >> TC [2] Compare w/ Previous Release
> >>
> >>
> https://ci.ignite.apache.org/viewLog.html?buildId=5909003=ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency=artifacts#
> >>
> >>
> >> The vote is formal, see voting guidelines
> >> https://www.apache.org/foundation/voting.html
> >>
> >> +1 - to accept Apache Ignite 2.10.0-rc2
> >> 0 - don't care either way
> >> -1 - DO NOT accept Apache Ignite Ignite 2.10.0-rc2 (explain why)
> >>
> >> See notes on how to verify release here
> >> https://www.apache.org/info/verification.html
> >> and
> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> >>
> >> This vote will be open until Sun Mar 14, 09:00 UTC.
> >> Please, write down the thread if you need additional time to check
> >> the release.
> >>
> >>
> https://www.timeanddate.com/countdown/vote?iso=20210314T12=166=Vote+for+the+Apache+Ignite+2.10.0+RC2=slab
> >>
>
>
>
>
Review Request: IGNITE-8635: Add a Method to Inspect BinaryObject Size
Hi All, I have raised a PR for the above mentioned issue. Please see and help review: https://github.com/apache/ignite/pull/8868 Regards, Atri -- Regards, Atri Apache Concerted
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Hi Val, Thanks for taking a look. I have updated the PR, please see and let me know your thoughts and feedback. Regards, Atri On Thu, Mar 11, 2021 at 6:16 AM Valentin Kulichenko wrote: > > Atri, > > I've added my comments to the PR. > > -Val > > On Wed, Mar 10, 2021 at 2:44 AM Atri Sharma wrote: > > > I have just updated the PR: > > > > https://github.com/apache/ignite/pull/8820 > > > > Please review. > > > > On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma wrote: > > > > > > Hi Val, > > > > > > Thank you for taking the time on this one. > > > > > > The main reason as to why I chose that signature was because I felt it > > > gave a clear idea of the interface that the user should expect when > > > using the method. So essentially, the user is providing a callback > > > listener himself/herself and the API is just using that to tie the > > > lifecycle of holding the semaphore. > > > > > > However, I do see your point and feel that the current signature that > > > the PR has will limit the usability of the method and make users jump > > > through more hoops. I have changed the signature to accept Callable > > > returning T. > > > > > > Regards, > > > > > > Atri > > > > > > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko > > > wrote: > > > > > > > > Hi Atri, > > > > > > > > First and foremost, we need to clarify the API for this functionality. > > > > There are two options presented in the ticket, but no clear decision > > about > > > > which one should be used. I personally think that the original > > signature > > > > (the one in the ticket description) makes more sense, but it looks like > > > > you've implemented an alternative suggested in the comments. What was > > your > > > > motivation behind that? > > > > > > > > As for where the method can be located, I'm OK if we add it directly > > to the > > > > IgniteSemaphore interface. > > > > > > > > -Val > > > > > > > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma wrote: > > > > > > > > > Gentle ping > > > > > > > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I would like to start a discussion around IGNITE-2399. > > > > > > > > > > > > Background: The typical use of IgniteSemaphore consists of > > acquiring > > > > > > the semaphore, performing the task and releasing the semaphore. > > This > > > > > > JIRA proposes a new method which will accept a callable, acquire > > the > > > > > > semaphore, and return a future. Upon completion of the future, the > > > > > > semaphore is released. > > > > > > > > > > > > This API seems useful for an easy encapsulation of the described > > use > > > > > > case and does not cause an internal flux since all the changes are > > > > > > focused at the public API. > > > > > > > > > > > > WIP PR for the same: > > > > > > > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > > > > > > > Please share your thoughts and comments. > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > Apache Concerted > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
I have just updated the PR: https://github.com/apache/ignite/pull/8820 Please review. On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma wrote: > > Hi Val, > > Thank you for taking the time on this one. > > The main reason as to why I chose that signature was because I felt it > gave a clear idea of the interface that the user should expect when > using the method. So essentially, the user is providing a callback > listener himself/herself and the API is just using that to tie the > lifecycle of holding the semaphore. > > However, I do see your point and feel that the current signature that > the PR has will limit the usability of the method and make users jump > through more hoops. I have changed the signature to accept Callable > returning T. > > Regards, > > Atri > > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko > wrote: > > > > Hi Atri, > > > > First and foremost, we need to clarify the API for this functionality. > > There are two options presented in the ticket, but no clear decision about > > which one should be used. I personally think that the original signature > > (the one in the ticket description) makes more sense, but it looks like > > you've implemented an alternative suggested in the comments. What was your > > motivation behind that? > > > > As for where the method can be located, I'm OK if we add it directly to the > > IgniteSemaphore interface. > > > > -Val > > > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma wrote: > > > > > Gentle ping > > > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, wrote: > > > > > > > Hi All, > > > > > > > > I would like to start a discussion around IGNITE-2399. > > > > > > > > Background: The typical use of IgniteSemaphore consists of acquiring > > > > the semaphore, performing the task and releasing the semaphore. This > > > > JIRA proposes a new method which will accept a callable, acquire the > > > > semaphore, and return a future. Upon completion of the future, the > > > > semaphore is released. > > > > > > > > This API seems useful for an easy encapsulation of the described use > > > > case and does not cause an internal flux since all the changes are > > > > focused at the public API. > > > > > > > > WIP PR for the same: > > > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > > > Please share your thoughts and comments. > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > -- > Regards, > > Atri > Apache Concerted -- Regards, Atri Apache Concerted
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Hi Val, Thank you for taking the time on this one. The main reason as to why I chose that signature was because I felt it gave a clear idea of the interface that the user should expect when using the method. So essentially, the user is providing a callback listener himself/herself and the API is just using that to tie the lifecycle of holding the semaphore. However, I do see your point and feel that the current signature that the PR has will limit the usability of the method and make users jump through more hoops. I have changed the signature to accept Callable returning T. Regards, Atri On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko wrote: > > Hi Atri, > > First and foremost, we need to clarify the API for this functionality. > There are two options presented in the ticket, but no clear decision about > which one should be used. I personally think that the original signature > (the one in the ticket description) makes more sense, but it looks like > you've implemented an alternative suggested in the comments. What was your > motivation behind that? > > As for where the method can be located, I'm OK if we add it directly to the > IgniteSemaphore interface. > > -Val > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma wrote: > > > Gentle ping > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, wrote: > > > > > Hi All, > > > > > > I would like to start a discussion around IGNITE-2399. > > > > > > Background: The typical use of IgniteSemaphore consists of acquiring > > > the semaphore, performing the task and releasing the semaphore. This > > > JIRA proposes a new method which will accept a callable, acquire the > > > semaphore, and return a future. Upon completion of the future, the > > > semaphore is released. > > > > > > This API seems useful for an easy encapsulation of the described use > > > case and does not cause an internal flux since all the changes are > > > focused at the public API. > > > > > > WIP PR for the same: > > > > > > https://github.com/apache/ignite/pull/8820 > > > > > > Please share your thoughts and comments. > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > -- Regards, Atri Apache Concerted
Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Gentle ping On Mon, 8 Mar 2021, 13:51 Atri Sharma, wrote: > Hi All, > > I would like to start a discussion around IGNITE-2399. > > Background: The typical use of IgniteSemaphore consists of acquiring > the semaphore, performing the task and releasing the semaphore. This > JIRA proposes a new method which will accept a callable, acquire the > semaphore, and return a future. Upon completion of the future, the > semaphore is released. > > This API seems useful for an easy encapsulation of the described use > case and does not cause an internal flux since all the changes are > focused at the public API. > > WIP PR for the same: > > https://github.com/apache/ignite/pull/8820 > > Please share your thoughts and comments. > > -- > Regards, > > Atri > Apache Concerted >
Re: Minor PR
Thank you! On Tue, 9 Mar 2021, 21:46 Ilya Kasnacheev, wrote: > Hello! > > You can use MTCGA to check those runs, such as > > https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull/8845/head=Latest > > This one looks OK, I will merge. > > Regards, > -- > Ilya Kasnacheev > > > пн, 8 мар. 2021 г. в 10:57, Atri Sharma : > > > Hello, > > > > I ran the test suite and here is the link: > > > > > > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_RunAll/5906237 > > > > I ran all of the failed tests but they passed for me. Most of them are > > marked as flaky. > > > > Regards, > > > > Atri > > > > On Fri, Mar 5, 2021 at 5:42 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > Please run tests against this change, get a green TC visa for the > ticket. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 3 мар. 2021 г. в 17:13, Atri Sharma : > > > > > > > Hi, > > > > > > > > Please help review this PR: > > > > > > > > https://github.com/apache/ignite/pull/8845 > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > >
IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore
Hi All, I would like to start a discussion around IGNITE-2399. Background: The typical use of IgniteSemaphore consists of acquiring the semaphore, performing the task and releasing the semaphore. This JIRA proposes a new method which will accept a callable, acquire the semaphore, and return a future. Upon completion of the future, the semaphore is released. This API seems useful for an easy encapsulation of the described use case and does not cause an internal flux since all the changes are focused at the public API. WIP PR for the same: https://github.com/apache/ignite/pull/8820 Please share your thoughts and comments. -- Regards, Atri Apache Concerted
Re: Minor PR
Hello, I ran the test suite and here is the link: https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_RunAll/5906237 I ran all of the failed tests but they passed for me. Most of them are marked as flaky. Regards, Atri On Fri, Mar 5, 2021 at 5:42 PM Ilya Kasnacheev wrote: > > Hello! > > Please run tests against this change, get a green TC visa for the ticket. > > Regards, > -- > Ilya Kasnacheev > > > ср, 3 мар. 2021 г. в 17:13, Atri Sharma : > > > Hi, > > > > Please help review this PR: > > > > https://github.com/apache/ignite/pull/8845 > > > > Regards, > > > > Atri > > -- Regards, Atri Apache Concerted
Review Request
Hi, Please help in reviewing: https://github.com/apache/ignite/pull/8820 Atri
Minor PR
Hi, Please help review this PR: https://github.com/apache/ignite/pull/8845 Regards, Atri
IGNITE-14225 -- New Helper Method For Tests
Hi, I have raised a new PR for IGNITE-14225: https://github.com/apache/ignite/pull/8833 Please help in reviewing. Regards, Atri -- Regards, Atri Apache Concerted
[jira] [Created] (IGNITE-14225) Add Helper Method to Check For Expected Exceptions in JUnitAssertAware
Atri Sharma created IGNITE-14225: Summary: Add Helper Method to Check For Expected Exceptions in JUnitAssertAware Key: IGNITE-14225 URL: https://issues.apache.org/jira/browse/IGNITE-14225 Project: Ignite Issue Type: Improvement Environment: We should have a helper method which allows checking if an expected exception is raised from a specific code block in tests Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: IGNITE-2399: Implement acquireAndExecute Utility Method
Thank you, very kind of you! Let me fix the PR and ping you back there. On Tue, 23 Feb 2021 at 2:05 PM, Данилов Семён wrote: > Hello Atri, > > I've read through your changes, have a look at the review. > > Kind regards, > Sam. > > 23.02.2021, 08:07, "Atri Sharma" : > > Hi All, > > > > Please help in reviewing the following PR: > > > > https://github.com/apache/ignite/pull/8820 > > > > Regards, > > > > Atri > > > > -- > > Regards, > > > > Atri > > Apache Concerted > -- Regards, Atri Apache Concerted
IGNITE-2399: Implement acquireAndExecute Utility Method
Hi All, Please help in reviewing the following PR: https://github.com/apache/ignite/pull/8820 Regards, Atri -- Regards, Atri Apache Concerted
Re: [MTCGA]: new failures in builds [5882649] needs to be handled
I ran the MVCC Test Suite 7 on branch with my changes a couple of times with no failures. On Sat, Feb 20, 2021 at 8:37 AM wrote: > > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and you > may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and fix > test failures or step down and some committer may revert you commit. > > *New Critical Failure in master-nightly MVCC Cache 7 > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccCache7?branch=%3Cdefault%3E > Changes may lead to failure were done by > - aleksey plekhanov > https://ci.ignite.apache.org/viewModification.html?modId=917288 > - atri sharma > https://ci.ignite.apache.org/viewModification.html?modId=917319 > - mikhail petrov > https://ci.ignite.apache.org/viewModification.html?modId=917314 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 06:06:54 20-02-2021 -- Regards, Atri Apache Concerted
JIRA Permissions
Hi, Please grant me JIRA permissions. My user id is atri -- Regards, Atri Apache Concerted
Re: IGNITE-12508
Thank you! Where do I figure out which tests are currently flaky? On Wed, Feb 17, 2021 at 8:58 PM Ilya Kasnacheev wrote: > > Hello! > > I think these tests are flaky, I have triggered them to re-run. > > Regards, > -- > Ilya Kasnacheev > > > ср, 17 февр. 2021 г. в 18:19, Atri Sharma : > > > Hello, > > > > Thanks for helping with this. According to the bot's comment on JIRA, > > the tests that are failing are: > > JdbcThinJdbcToCacheDataTypesCoverageTest.testTimeDataType > > IgnitePdsWithIndexingCoreTestSuite: > > WalRecoveryTxLogicalRecordsTest.testRecoveryNoPageLost3 > > > > I have verified that these tests pass locally for me (beasted them) > > and also manually debugged to see if the new method was giving the > > correct behaviour. > > > > Please advice. > > > > Atri > > > > On Wed, Feb 17, 2021 at 2:37 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > Thank you! I have started a MTCGA run since there were not any. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 17 февр. 2021 г. в 07:49, Atri Sharma : > > > > > > > Thank you! I have updated the PR per your comments. > > > > > > > > On Tue, Feb 16, 2021 at 9:29 PM Ilya Kasnacheev > > > > wrote: > > > > > > > > > > Hello! > > > > > > > > > > I have left some comments to the PR. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > пн, 15 февр. 2021 г. в 18:55, Atri Sharma : > > > > > > > > > > > Hi all, > > > > > > > > > > > > I have raised a PR for IGNITE-12508: > > > > > > > > > > > > https://github.com/apache/ignite/pull/8802 > > > > > > > > > > > > Requesting inputs and feedback on the same, please. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Atri > > > > > > Apache Concerted > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: IGNITE-12508
Hello, Thanks for helping with this. According to the bot's comment on JIRA, the tests that are failing are: JdbcThinJdbcToCacheDataTypesCoverageTest.testTimeDataType IgnitePdsWithIndexingCoreTestSuite: WalRecoveryTxLogicalRecordsTest.testRecoveryNoPageLost3 I have verified that these tests pass locally for me (beasted them) and also manually debugged to see if the new method was giving the correct behaviour. Please advice. Atri On Wed, Feb 17, 2021 at 2:37 PM Ilya Kasnacheev wrote: > > Hello! > > Thank you! I have started a MTCGA run since there were not any. > > Regards, > -- > Ilya Kasnacheev > > > ср, 17 февр. 2021 г. в 07:49, Atri Sharma : > > > Thank you! I have updated the PR per your comments. > > > > On Tue, Feb 16, 2021 at 9:29 PM Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > I have left some comments to the PR. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > пн, 15 февр. 2021 г. в 18:55, Atri Sharma : > > > > > > > Hi all, > > > > > > > > I have raised a PR for IGNITE-12508: > > > > > > > > https://github.com/apache/ignite/pull/8802 > > > > > > > > Requesting inputs and feedback on the same, please. > > > > > > > > Regards, > > > > > > > > Atri > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
Re: IGNITE-12508
Thank you! I have updated the PR per your comments. On Tue, Feb 16, 2021 at 9:29 PM Ilya Kasnacheev wrote: > > Hello! > > I have left some comments to the PR. > > Regards, > -- > Ilya Kasnacheev > > > пн, 15 февр. 2021 г. в 18:55, Atri Sharma : > > > Hi all, > > > > I have raised a PR for IGNITE-12508: > > > > https://github.com/apache/ignite/pull/8802 > > > > Requesting inputs and feedback on the same, please. > > > > Regards, > > > > Atri > > > > -- > > Regards, > > > > Atri > > Apache Concerted > > -- Regards, Atri Apache Concerted
IGNITE-12508
Hi all, I have raised a PR for IGNITE-12508: https://github.com/apache/ignite/pull/8802 Requesting inputs and feedback on the same, please. Regards, Atri -- Regards, Atri Apache Concerted
