Re: Github pull requests are not linked automatically to jira tickets.
Pavel, Thank you for starting this discussion! Controlling all this repo options on our side sounds very attractive. Personally I do not have any arguments about the proposal. Best regards, Ivan Pavlukhin чт, 7 мая 2020 г. в 17:39, Pavel Pereslegin : > > Igniters, > > recent github pull requests are not automatically linked to the jira > tickets. > > Infra advises creating a yaml configuration in the root of the repository > with the settings for the github bot [1]. > Examples of such configuration in Hive [2] and HBase [3]. > I suggest the following settings for Ignite [4]. > > If there are no objections, I'll file a ticket for this. > > [1] https://issues.apache.org/jira/browse/INFRA-20177 > [2] https://github.com/apache/hbase/blob/master/.asf.yaml > [3] https://github.com/apache/hive/blob/master/.asf.yaml > [4] https://gist.github.com/xtern/572cbc7d4a7916f6e933fbafe5034492
Re: [DISCUSSION] ASF Board Report Draft, May 2020
Hi Dmitry, thanks! Please add the following to the project activity section: - Released a new version of the Apache Ignite website: https://ignite.apache.org - Prepared a public roadmap with significant improvements planned for the rest of 2020: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap - Flume, flinks and some other extensions have been migrated to the extensions repository: https://github.com/apache/ignite-extensions/tree/master/modules - Community members gave 19 public online and offline presentations about the project since the beginning of 2020. Around 8 talks are yet to be presented and even more to follow: https://ignite.apache.org/events.html To my knowledge, the Spring Boot extension has been already released ( https://apacheignite-mix.readme.io/docs/spring-boot). @Nikolay Izhikov , is it ready to be announced? Also, I would suggest using this as a definition of the project (what Ignite is) - Apache Ignite® is a horizontally scalable, fault-tolerant distributed in-memory computing platform for building real-time applications that can process terabytes of data with in-memory speed. - Denis On Thu, May 7, 2020 at 7:46 AM Dmitriy Pavlov wrote: > Hi Igniters, > > To increase involvement of each member of the community and increase > transparency I'd like to share report draft for ASF board with all dev@ > community. Thanks to Ivan P. for this idea. > > What would you like to add to this report? Major features, events, popular > publications could be mentioned in the report (main manual sections are > community and project activity, and issues - if any). Let's make it more > representative and useful. I'm going to publish it during weekend. Please > find report draft below and share your thoughts. > > Sincerely, > Dmitriy Pavlov > > ## Description: > The mission of Ignite is the creation and maintenance of software related > to > High-performance, integrated and distributed in-memory platform for > computing > and transacting on large-scale data sets in real-time > > ## Issues: > There are no issues requiring board attention. > > ## Membership Data: > Apache Ignite was founded 2015-08-19 (5 years ago) > There are currently 50 committers and 34 PMC members in this project. > The Committer-to-PMC ratio is roughly 7:5. > > Community changes, past quarter: > - Maxim Muzafarov was added to the PMC on 2020-04-28 > - Slava Koptilin was added as committer on 2020-02-15 > > ## Project Activity: > 2.8.0 was released on 2020-03-03. > Spring boot starter was voted to be released. > Apache Ignite 2.8.1 release is under preparation. > > ## Community Health: > Activity at development related mailing lists (dev@, issues@, > notifications@) > > increased in compare to past quarter (+30..40%) > Commits count this quarter is slightly less (-17%) >
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Maxim, congrats! Great job, respect! чт, 7 мая 2020 г. в 23:04, Vyacheslav Daradur : > Great job! My congratulations! > > чт, 7 мая 2020 г. в 16:09, Alex Plehanov : > > > Maxim, congratulations! > > > > > > чт, 7 мая 2020 г. в 15:12, Denis Garus : > > > > > Maxim, Congrats! > > > Great job! > > > > > > чт, 7 мая 2020 г. в 15:02, Nikita Amelchev : > > > > > > > Maxim, congrats! > > > > > > > > чт, 7 мая 2020 г. в 14:55, Nikolay Izhikov : > > > > > > > > > > Congrats. > > > > > > > > > > > 7 мая 2020 г., в 14:54, Ivan Pavlukhin > > > > написал(а): > > > > > > > > > > > > Maxim, > > > > > > > > > > > > My congratulations! Well deserved! > > > > > > > > > > > > P.S. We should mention snapshots for persistent caches in the > > > > > > achievement list. Great job! > > > > > > > > > > > > Best regards, > > > > > > Ivan Pavlukhin > > > > > > > > > > > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > > > > > >> > > > > > >> The Project Management Committee (PMC) for Apache Ignite > > > > > >> > > > > > >> has invited Maxim Muzafarov to become new PMC member and we are > > > > pleased to > > > > > >> announce that he has accepted. > > > > > >> > > > > > >> > > > > > >> > > > > > >> Maxim is active dev list participant, speaker at meetups, and > > > > contributes > > > > > >> to additional checks of the product using travis and started new > > > file > > > > > >> rebalance. Maxim did a fantastic job to make release 2.8 > possible > > > > > >> > > > > > >> > > > > > >> > > > > > >> Being a PMC member enables assistance with the management > > > > > >> > > > > > >> and to guide the direction of the project. > > > > > >> > > > > > >> Maxim, congrats with new role and keep the pace ! > > > > > >> > > > > > >> > > > > > >> > > > > > >> Best Regards, > > > > > >> > > > > > >> Dmitriy Pavlov > > > > > >> > > > > > >> on behalf of Apache Ignite PMC > > > > > > > > > > > > > > > > > -- > > > > Best wishes, > > > > Amelchev Nikita > > > > > > > > > > -- > Best Regards, > Vyacheslav D. >
[jira] [Created] (IGNITE-12991) Calcite integration. Pass cancel flag to VolcanoPlanner
Igor Seliverstov created IGNITE-12991:
-
Summary: Calcite integration. Pass cancel flag to VolcanoPlanner
Key: IGNITE-12991
URL: https://issues.apache.org/jira/browse/IGNITE-12991
Project: Ignite
Issue Type: Improvement
Reporter: Igor Seliverstov
see {{AbstractRelOptPlanner.java:91}}, here {{CancelFlag}} is used to cancel
planning loop. We need to pass it into a newly created context and bind its
state with {{PlanningContext#queryCancel}} to break possible infinite planning
loop. See also {{PlanningContext#unwrap}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Great job! My congratulations! чт, 7 мая 2020 г. в 16:09, Alex Plehanov : > Maxim, congratulations! > > > чт, 7 мая 2020 г. в 15:12, Denis Garus : > > > Maxim, Congrats! > > Great job! > > > > чт, 7 мая 2020 г. в 15:02, Nikita Amelchev : > > > > > Maxim, congrats! > > > > > > чт, 7 мая 2020 г. в 14:55, Nikolay Izhikov : > > > > > > > > Congrats. > > > > > > > > > 7 мая 2020 г., в 14:54, Ivan Pavlukhin > > > написал(а): > > > > > > > > > > Maxim, > > > > > > > > > > My congratulations! Well deserved! > > > > > > > > > > P.S. We should mention snapshots for persistent caches in the > > > > > achievement list. Great job! > > > > > > > > > > Best regards, > > > > > Ivan Pavlukhin > > > > > > > > > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > > > > >> > > > > >> The Project Management Committee (PMC) for Apache Ignite > > > > >> > > > > >> has invited Maxim Muzafarov to become new PMC member and we are > > > pleased to > > > > >> announce that he has accepted. > > > > >> > > > > >> > > > > >> > > > > >> Maxim is active dev list participant, speaker at meetups, and > > > contributes > > > > >> to additional checks of the product using travis and started new > > file > > > > >> rebalance. Maxim did a fantastic job to make release 2.8 possible > > > > >> > > > > >> > > > > >> > > > > >> Being a PMC member enables assistance with the management > > > > >> > > > > >> and to guide the direction of the project. > > > > >> > > > > >> Maxim, congrats with new role and keep the pace ! > > > > >> > > > > >> > > > > >> > > > > >> Best Regards, > > > > >> > > > > >> Dmitriy Pavlov > > > > >> > > > > >> on behalf of Apache Ignite PMC > > > > > > > > > > > > > -- > > > Best wishes, > > > Amelchev Nikita > > > > > > -- Best Regards, Vyacheslav D.
Ignite Releases Plan
Igniters, Thanks for helping to put together our first roadmap for the rest of 2020 [1]. Turned out to be a handy source that should be appreciated by Ignite application developers. By looking at the page, it feels like we can plan a couple of releases: - Ignite 2.9 (early September) - it includes all the improvements that should be ready throughout May-July. We reserve August for issues fixing and final release steps. - Ignite 2.10 (late January 2021) - the release is for features that should be completed within the August-November timeframe. Considering the holiday season, we'll use December and most of January for final release procedures. What do you think? Also, is there anybody who is ready to take over release management tasks for 2.9? [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap - Denis
Re: [DISCUSSION] Pull-request checks on GitHub
Ivan,
This is the default configuration of travis-ci. Nothing was changed by me here.
Please, correct me if I'm wrong. According to my knowledge the option
mentioned by you have the following meaning:
- check `continuous-integration/travis-ci/pr` -- travis will merge
changes to the master branch (locally) and build merged changes.
- check `continuous-integration/travis-ci/push` -- travis will build
the pr branch only.
On Thu, 7 May 2020 at 21:43, Maxim Muzafarov wrote:
>
> Folks,
>
>
> I've updated the checklist according to your suggestions [1].
>
> Added.
> + Treat PR title as the final squashed commit message.
> + The description explains what and why vs. how
>
> Removed.
> - Commits have the following pattern
>
>
> [1] https://github.com/apache/ignite/pull/7765/files
>
> On Mon, 4 May 2020 at 10:41, Ivan Pavlukhin wrote:
> >
> > Pavel,
> >
> > > I think this thread is a good opportunity to discuss commit message
> > > guidelines.
> >
> > We had a thread about it recently [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/rde6e8258537704433286a60e1d0142485c25228a46561544d35b9704%40%3Cdev.ignite.apache.org%3E
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > пн, 4 мая 2020 г. в 10:38, Ivan Pavlukhin :
> > >
> > > Maxim,
> > >
> > > Thanks again for doing great things!
> > >
> > > Out of curiosity, could you please shed a light why there are 2 travis
> > > checks for PR [1]? I am about checks named
> > > continuous-integration/travis-ci/pr and
> > > continuous-integration/travis-ci/push.
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> > > вс, 3 мая 2020 г. в 13:08, Pavel Tupitsyn :
> > > >
> > > > Igniters, Maxim,
> > > >
> > > > I think this thread is a good opportunity to discuss commit message
> > > > guidelines.
> > > > I suggest the following:
> > > >
> > > > 1. Treat PR title + description as the final squashed commit message.
> > > > PR author is responsible for writing that properly.
> > > > Committer who merges the PR is responsible for validating that and using
> > > > that for the actual squash commit.
> > > >
> > > > 2. Adopt the following Git commit message rules (partially from
> > > > https://chris.beams.io/posts/git-commit/):
> > > > - Start with IGNITE-
> > > > - Use imperative mood in the subject line ("Fix foobar crash on start",
> > > > "Add baz metric")
> > > > - Capitalize the subject line
> > > > - Do not end the subject line with a period
> > > > - Use the body to explain what and why vs. how
> > > >
> > > > Thoughts?
> > > >
> > > > On Sun, May 3, 2020 at 11:53 AM Maxim Muzafarov
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I have the following in my mind:
> > > > > 1. This checklist is for discussion and may be changed.
> > > > > 2. Commits can be squashed in the branch prior to asking a review, but
> > > > > when the review is in progress a good naming may help to understand
> > > > > the changes.
> > > > > 3. It's true that the commit message can be changed prior to merging
> > > > > the master branch, but it's better to merge the PR with an initial
> > > > > authored commit message `as is`.
> > > > >
> > > > > On Sat, 2 May 2020 at 18:20, Guru Stron
> > > > > wrote:
> > > > > >
> > > > > > Maxim,
> > > > > >
> > > > > > I have a small question about "Commits have the following
> > > > > > pattern..". Is
> > > > > > it really needed cause AFAIK commits in the PR are squashed. Or am
> > > > > > I
> > > > > > missing something?
> > > > > >
> > > > > > On Thu, Apr 30, 2020, 8:33 PM Maxim Muzafarov
> > > > > > wrote:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > >
> > > > > > > I've created the pull request template for GitHub.
> > > > > > > Please, take a look and let me know what you think [1] [2].
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > https://github.com/apache/ignite/pull/7765/files#diff-195a635ad245ded9076330e31134bd80
> > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12937
> > > > > > >
> > > > > > > On Sun, 26 Apr 2020 at 20:35, Saikat Maitra
> > > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Maxim,
> > > > > > > >
> > > > > > > > Thank you for enabling travis ci in ignite repo. It is very
> > > > > > > > helpful
> > > > > to
> > > > > > > see
> > > > > > > > PR build results integrated in PR request.
> > > > > > > >
> > > > > > > > I will enable it in ignite-extensions repo as well.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Saikat
> > > > > > > >
> > > > > > > > On Mon, Apr 20, 2020 at 12:14 PM Pavel Tupitsyn <
> > > > > ptupit...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Maxim, pull request template is a great idea.
> > > > > > > > > We can put a checklist there along with the links to the
> > > > > guidelines,
> > > > > > > > > something like this:
> > > > > > > > >
> > > > > > > > > [ ] Coding Guidelines are followed
> > > > > > > > > [ ] TeamCity build passes
> > > > > > > > > [ ] JIRA ticked is in
Re: [DISCUSSION] Pull-request checks on GitHub
Folks,
I've updated the checklist according to your suggestions [1].
Added.
+ Treat PR title as the final squashed commit message.
+ The description explains what and why vs. how
Removed.
- Commits have the following pattern
[1] https://github.com/apache/ignite/pull/7765/files
On Mon, 4 May 2020 at 10:41, Ivan Pavlukhin wrote:
>
> Pavel,
>
> > I think this thread is a good opportunity to discuss commit message
> > guidelines.
>
> We had a thread about it recently [1].
>
> [1]
> https://lists.apache.org/thread.html/rde6e8258537704433286a60e1d0142485c25228a46561544d35b9704%40%3Cdev.ignite.apache.org%3E
>
> Best regards,
> Ivan Pavlukhin
>
> пн, 4 мая 2020 г. в 10:38, Ivan Pavlukhin :
> >
> > Maxim,
> >
> > Thanks again for doing great things!
> >
> > Out of curiosity, could you please shed a light why there are 2 travis
> > checks for PR [1]? I am about checks named
> > continuous-integration/travis-ci/pr and
> > continuous-integration/travis-ci/push.
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > вс, 3 мая 2020 г. в 13:08, Pavel Tupitsyn :
> > >
> > > Igniters, Maxim,
> > >
> > > I think this thread is a good opportunity to discuss commit message
> > > guidelines.
> > > I suggest the following:
> > >
> > > 1. Treat PR title + description as the final squashed commit message.
> > > PR author is responsible for writing that properly.
> > > Committer who merges the PR is responsible for validating that and using
> > > that for the actual squash commit.
> > >
> > > 2. Adopt the following Git commit message rules (partially from
> > > https://chris.beams.io/posts/git-commit/):
> > > - Start with IGNITE-
> > > - Use imperative mood in the subject line ("Fix foobar crash on start",
> > > "Add baz metric")
> > > - Capitalize the subject line
> > > - Do not end the subject line with a period
> > > - Use the body to explain what and why vs. how
> > >
> > > Thoughts?
> > >
> > > On Sun, May 3, 2020 at 11:53 AM Maxim Muzafarov wrote:
> > >
> > > > Hello,
> > > >
> > > > I have the following in my mind:
> > > > 1. This checklist is for discussion and may be changed.
> > > > 2. Commits can be squashed in the branch prior to asking a review, but
> > > > when the review is in progress a good naming may help to understand
> > > > the changes.
> > > > 3. It's true that the commit message can be changed prior to merging
> > > > the master branch, but it's better to merge the PR with an initial
> > > > authored commit message `as is`.
> > > >
> > > > On Sat, 2 May 2020 at 18:20, Guru Stron
> > > > wrote:
> > > > >
> > > > > Maxim,
> > > > >
> > > > > I have a small question about "Commits have the following pattern..".
> > > > > Is
> > > > > it really needed cause AFAIK commits in the PR are squashed. Or am I
> > > > > missing something?
> > > > >
> > > > > On Thu, Apr 30, 2020, 8:33 PM Maxim Muzafarov
> > > > > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >
> > > > > > I've created the pull request template for GitHub.
> > > > > > Please, take a look and let me know what you think [1] [2].
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > https://github.com/apache/ignite/pull/7765/files#diff-195a635ad245ded9076330e31134bd80
> > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12937
> > > > > >
> > > > > > On Sun, 26 Apr 2020 at 20:35, Saikat Maitra
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Maxim,
> > > > > > >
> > > > > > > Thank you for enabling travis ci in ignite repo. It is very
> > > > > > > helpful
> > > > to
> > > > > > see
> > > > > > > PR build results integrated in PR request.
> > > > > > >
> > > > > > > I will enable it in ignite-extensions repo as well.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Saikat
> > > > > > >
> > > > > > > On Mon, Apr 20, 2020 at 12:14 PM Pavel Tupitsyn <
> > > > ptupit...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Maxim, pull request template is a great idea.
> > > > > > > > We can put a checklist there along with the links to the
> > > > guidelines,
> > > > > > > > something like this:
> > > > > > > >
> > > > > > > > [ ] Coding Guidelines are followed
> > > > > > > > [ ] TeamCity build passes
> > > > > > > > [ ] JIRA ticked is in Patch Available state, review has been
> > > > requested
> > > > > > in
> > > > > > > > comments
> > > > > > > > [ ] Something else?
> > > > > > > >
> > > > > > > > On Mon, Apr 20, 2020 at 8:09 PM Maxim Muzafarov
> > > > > > > > > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Pavel,
> > > > > > > > >
> > > > > > > > > Sorry for the incomplete message.
> > > > > > > > >
> > > > > > > > > 2. I mentioned it too. Hopefully, builds > 4 hrs would not be
> > > > > > > > > too
> > > > > > often.
> > > > > > > > > The reason - there are limited job-workers shared between all
> > > > > > > > > the
> > > > > > > > > Apache projects. I found some details here [1] [2].
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > >
Re: [DISCUSS] Remove TC badge from README.md
IMO, we can remove. This badge does not work since GitHub started to cache images. Now TC's status .SVG does not work properly. чт, 7 мая 2020 г. в 09:25, Ivan Pavlukhin : > Folks, > > I created a ticket [1] and prepared PR with magic number =). > > Please review. > > [1] https://issues.apache.org/jira/browse/IGNITE-12987 > > Best regards, > Ivan Pavlukhin > > пн, 4 мая 2020 г. в 10:01, Ivan Pavlukhin : > > > > Hi Igniters, > > > > Inspired by a neighbor thread about PR checks [1]. It brought my > > attention that now we have a neat Travis badge and a strange TC badge > > which is not very useful from a first glance. > > > > What do you think, is it ought to completely remove TC badge from readme? > > > > [1] > https://lists.apache.org/thread.html/r1580e2fb23728e83afbe1bad2b4cf7e9cece0e19a1d305e5755d7dd1%40%3Cdev.ignite.apache.org%3E > > > > Best regards, > > Ivan Pavlukhin >
Re: IEP-44 Thin Client Discovery
Igniters, the feature is ready for review: https://issues.apache.org/jira/browse/IGNITE-12932 https://github.com/apache/ignite/pull/7744 On Thu, May 7, 2020 at 6:23 PM Pavel Tupitsyn wrote: > Alex, > > Event subscription is a good idea. We should certainly add this in future. > However, it might be non-trivial with multiple connections: should we > subscribe on every server? > Then we'll get an event from every server, which seems redundant. > > Igor, > > Agreed. Let's keep existing behavior. > > On Thu, May 7, 2020 at 5:29 PM Igor Sapego wrote: > >> Pavel, >> >> First of all, I think we need to move it out of scope of this feature. >> >> Also, why do we need to keep connection alive? I think, if user do not >> use connection for a long time and connection is lost we could just >> detect this and re-establish connection on the next user action which >> requires network interaction. Any issues with this approach? >> >> Best Regards, >> Igor >> >> >> On Thu, May 7, 2020 at 1:18 AM Alex Plehanov >> wrote: >> >> > Pavel, >> > >> > Since we have a notification mechanism for thin clients now, we can >> > implement a subscription to some types of events and this can be used >> > to inform a client about topology change as well. I think it's a >> > more appropriate way to detect topology changes than ping requests. But >> > approach with ping requests has another advantage: the client can detect >> > that connection was lost earlier. With subscriptions approach client >> will >> > detect problems with a connection only after the next request to the >> > server. >> > >> > >> > ср, 6 мая 2020 г. в 17:31, Pavel Tupitsyn : >> > >> > > Igniters, let's discuss the following issue: >> > > >> > > For partition awareness, and now for cluster discovery, we use a >> response >> > > flag to detect topology changes. >> > > The problem is - if the client does not do anything (user code does >> not >> > > perform operations), >> > > then we'll never know about topology changes and may even lose the >> > cluster >> > > (all known nodes leave). >> > > >> > > Should we introduce some keep-alive mechanism, so that thin clients >> send >> > > periodic ping requests? >> > > Maybe do this as a separate feature. >> > > >> > > Thoughts? >> > > >> > > On Tue, Apr 28, 2020 at 8:14 PM Pavel Tupitsyn >> > > wrote: >> > > >> > > > Ok, I've updated IEP and POC accordingly: >> > > > * Config flag removed >> > > > * IPs and host names retrieval simplified - use existing node >> > properties >> > > > and attributes instead of Compute >> > > > >> > > > On Tue, Apr 28, 2020 at 7:57 PM Igor Sapego >> > wrote: >> > > > >> > > >> I guess it makes sense. If anyone needs more control over >> connection >> > > >> we would need to implement a new feature anyway (like node filter >> we >> > > >> discussed earlier) >> > > >> >> > > >> Best Regards, >> > > >> Igor >> > > >> >> > > >> >> > > >> On Tue, Apr 28, 2020 at 12:29 PM Pavel Tupitsyn < >> ptupit...@apache.org >> > > >> > > >> wrote: >> > > >> >> > > >> > > enable the capability if the best effort affinity is on >> > > >> > I agree, makes sense. >> > > >> > >> > > >> > Igor, what do you think? >> > > >> > >> > > >> > On Tue, Apr 28, 2020 at 8:25 AM Denis Magda >> > > wrote: >> > > >> > >> > > >> > > Pavel, >> > > >> > > >> > > >> > > That would be a tremendous improvement for the recently release >> > best >> > > >> > effort >> > > >> > > affinity feature. Without this capability, we force application >> > > >> > developers >> > > >> > > to reopen thin client connections every type a cluster is >> scaled >> > > out. >> > > >> I >> > > >> > > believe that once the folks start using the best effort >> affinity, >> > > >> we'll >> > > >> > be >> > > >> > > hearing more of a feature request for what you're proposing in >> > this >> > > >> > thread. >> > > >> > > So, thanks for taking care of this proactively! >> > > >> > > >> > > >> > > As for the public API changes, do we really need any extra >> flag? I >> > > >> would >> > > >> > > enable the capability if the best effort affinity is on. For >> me, >> > > it's >> > > >> > just >> > > >> > > a natural improvement of the latter and it sounds reasonable to >> > > reuse >> > > >> the >> > > >> > > best effort affinity's flag. >> > > >> > > >> > > >> > > - >> > > >> > > Denis >> > > >> > > >> > > >> > > >> > > >> > > On Mon, Apr 27, 2020 at 2:58 AM Pavel Tupitsyn < >> > > ptupit...@apache.org> >> > > >> > > wrote: >> > > >> > > >> > > >> > > > Igniters, >> > > >> > > > >> > > >> > > > I've prepared an IEP [1] and a POC [2] for Thin Client >> Discovery >> > > >> > feature. >> > > >> > > > Let's discuss it here. >> > > >> > > > >> > > >> > > > In particular, I'd like to address the following points: >> > > >> > > > >> > > >> > > > 1. Value: do you think this would be a good feature to have? >> > > >> > > > 2. Public API changes: is a boolean property enough? Should >> we >> > > have >> > > >> > > > something more complex, so users can plug in custom
Re: IEP-44 Thin Client Discovery
Alex, Event subscription is a good idea. We should certainly add this in future. However, it might be non-trivial with multiple connections: should we subscribe on every server? Then we'll get an event from every server, which seems redundant. Igor, Agreed. Let's keep existing behavior. On Thu, May 7, 2020 at 5:29 PM Igor Sapego wrote: > Pavel, > > First of all, I think we need to move it out of scope of this feature. > > Also, why do we need to keep connection alive? I think, if user do not > use connection for a long time and connection is lost we could just > detect this and re-establish connection on the next user action which > requires network interaction. Any issues with this approach? > > Best Regards, > Igor > > > On Thu, May 7, 2020 at 1:18 AM Alex Plehanov > wrote: > > > Pavel, > > > > Since we have a notification mechanism for thin clients now, we can > > implement a subscription to some types of events and this can be used > > to inform a client about topology change as well. I think it's a > > more appropriate way to detect topology changes than ping requests. But > > approach with ping requests has another advantage: the client can detect > > that connection was lost earlier. With subscriptions approach client will > > detect problems with a connection only after the next request to the > > server. > > > > > > ср, 6 мая 2020 г. в 17:31, Pavel Tupitsyn : > > > > > Igniters, let's discuss the following issue: > > > > > > For partition awareness, and now for cluster discovery, we use a > response > > > flag to detect topology changes. > > > The problem is - if the client does not do anything (user code does not > > > perform operations), > > > then we'll never know about topology changes and may even lose the > > cluster > > > (all known nodes leave). > > > > > > Should we introduce some keep-alive mechanism, so that thin clients > send > > > periodic ping requests? > > > Maybe do this as a separate feature. > > > > > > Thoughts? > > > > > > On Tue, Apr 28, 2020 at 8:14 PM Pavel Tupitsyn > > > wrote: > > > > > > > Ok, I've updated IEP and POC accordingly: > > > > * Config flag removed > > > > * IPs and host names retrieval simplified - use existing node > > properties > > > > and attributes instead of Compute > > > > > > > > On Tue, Apr 28, 2020 at 7:57 PM Igor Sapego > > wrote: > > > > > > > >> I guess it makes sense. If anyone needs more control over connection > > > >> we would need to implement a new feature anyway (like node filter we > > > >> discussed earlier) > > > >> > > > >> Best Regards, > > > >> Igor > > > >> > > > >> > > > >> On Tue, Apr 28, 2020 at 12:29 PM Pavel Tupitsyn < > ptupit...@apache.org > > > > > > >> wrote: > > > >> > > > >> > > enable the capability if the best effort affinity is on > > > >> > I agree, makes sense. > > > >> > > > > >> > Igor, what do you think? > > > >> > > > > >> > On Tue, Apr 28, 2020 at 8:25 AM Denis Magda > > > wrote: > > > >> > > > > >> > > Pavel, > > > >> > > > > > >> > > That would be a tremendous improvement for the recently release > > best > > > >> > effort > > > >> > > affinity feature. Without this capability, we force application > > > >> > developers > > > >> > > to reopen thin client connections every type a cluster is scaled > > > out. > > > >> I > > > >> > > believe that once the folks start using the best effort > affinity, > > > >> we'll > > > >> > be > > > >> > > hearing more of a feature request for what you're proposing in > > this > > > >> > thread. > > > >> > > So, thanks for taking care of this proactively! > > > >> > > > > > >> > > As for the public API changes, do we really need any extra > flag? I > > > >> would > > > >> > > enable the capability if the best effort affinity is on. For me, > > > it's > > > >> > just > > > >> > > a natural improvement of the latter and it sounds reasonable to > > > reuse > > > >> the > > > >> > > best effort affinity's flag. > > > >> > > > > > >> > > - > > > >> > > Denis > > > >> > > > > > >> > > > > > >> > > On Mon, Apr 27, 2020 at 2:58 AM Pavel Tupitsyn < > > > ptupit...@apache.org> > > > >> > > wrote: > > > >> > > > > > >> > > > Igniters, > > > >> > > > > > > >> > > > I've prepared an IEP [1] and a POC [2] for Thin Client > Discovery > > > >> > feature. > > > >> > > > Let's discuss it here. > > > >> > > > > > > >> > > > In particular, I'd like to address the following points: > > > >> > > > > > > >> > > > 1. Value: do you think this would be a good feature to have? > > > >> > > > 2. Public API changes: is a boolean property enough? Should we > > > have > > > >> > > > something more complex, so users can plug in custom logic to > > > filter > > > >> > > and/or > > > >> > > > translate IPs and host names? > > > >> > > > 3. Server-side implementation details: should we use Compute, > > Node > > > >> > > > Attributes, or something else to retrieve client endpoints > from > > > all > > > >> > nodes > > > >> > > > in cluster? > > > >> > > > > > > >> > > > [1] > > > >> > > > > > > >> >
[ANNOUNCE] Apache Ignite Spring Boot extensions 1.0.0 Released
The Apache Ignite Community is pleased to announce the release of Apache Ignite Spring Boot extensions 1.0.0 Apache Ignite [1] is a memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale. This is the very first release of spring-boot extensions: https://ignite.apache.org/releases/ext/spring-boot-1.0.0/release_notes.html Download the latest Apache Ignite Spring Boot extensions version from here: * https://repo.maven.apache.org/maven2/org/apache/ignite/ignite-spring-boot-autoconfigure-ext/1.0.0/ * https://repo.maven.apache.org/maven2/org/apache/ignite/ignite-spring-boot-thin-client-autoconfigure-ext/1.0.0/ Please let us know [2] if you encounter any problems. Regards, Nikolay Izhikov on behalf of Apache Ignite community [1] https://ignite.apache.org [2] https://ignite.apache.org/community/resources.html#ask
[DISCUSSION] ASF Board Report Draft, May 2020
Hi Igniters, To increase involvement of each member of the community and increase transparency I'd like to share report draft for ASF board with all dev@ community. Thanks to Ivan P. for this idea. What would you like to add to this report? Major features, events, popular publications could be mentioned in the report (main manual sections are community and project activity, and issues - if any). Let's make it more representative and useful. I'm going to publish it during weekend. Please find report draft below and share your thoughts. Sincerely, Dmitriy Pavlov ## Description: The mission of Ignite is the creation and maintenance of software related to High-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time ## Issues: There are no issues requiring board attention. ## Membership Data: Apache Ignite was founded 2015-08-19 (5 years ago) There are currently 50 committers and 34 PMC members in this project. The Committer-to-PMC ratio is roughly 7:5. Community changes, past quarter: - Maxim Muzafarov was added to the PMC on 2020-04-28 - Slava Koptilin was added as committer on 2020-02-15 ## Project Activity: 2.8.0 was released on 2020-03-03. Spring boot starter was voted to be released. Apache Ignite 2.8.1 release is under preparation. ## Community Health: Activity at development related mailing lists (dev@, issues@, notifications@) increased in compare to past quarter (+30..40%) Commits count this quarter is slightly less (-17%)
Github pull requests are not linked automatically to jira tickets.
Igniters, recent github pull requests are not automatically linked to the jira tickets. Infra advises creating a yaml configuration in the root of the repository with the settings for the github bot [1]. Examples of such configuration in Hive [2] and HBase [3]. I suggest the following settings for Ignite [4]. If there are no objections, I'll file a ticket for this. [1] https://issues.apache.org/jira/browse/INFRA-20177 [2] https://github.com/apache/hbase/blob/master/.asf.yaml [3] https://github.com/apache/hive/blob/master/.asf.yaml [4] https://gist.github.com/xtern/572cbc7d4a7916f6e933fbafe5034492
Re: IEP-44 Thin Client Discovery
Pavel, First of all, I think we need to move it out of scope of this feature. Also, why do we need to keep connection alive? I think, if user do not use connection for a long time and connection is lost we could just detect this and re-establish connection on the next user action which requires network interaction. Any issues with this approach? Best Regards, Igor On Thu, May 7, 2020 at 1:18 AM Alex Plehanov wrote: > Pavel, > > Since we have a notification mechanism for thin clients now, we can > implement a subscription to some types of events and this can be used > to inform a client about topology change as well. I think it's a > more appropriate way to detect topology changes than ping requests. But > approach with ping requests has another advantage: the client can detect > that connection was lost earlier. With subscriptions approach client will > detect problems with a connection only after the next request to the > server. > > > ср, 6 мая 2020 г. в 17:31, Pavel Tupitsyn : > > > Igniters, let's discuss the following issue: > > > > For partition awareness, and now for cluster discovery, we use a response > > flag to detect topology changes. > > The problem is - if the client does not do anything (user code does not > > perform operations), > > then we'll never know about topology changes and may even lose the > cluster > > (all known nodes leave). > > > > Should we introduce some keep-alive mechanism, so that thin clients send > > periodic ping requests? > > Maybe do this as a separate feature. > > > > Thoughts? > > > > On Tue, Apr 28, 2020 at 8:14 PM Pavel Tupitsyn > > wrote: > > > > > Ok, I've updated IEP and POC accordingly: > > > * Config flag removed > > > * IPs and host names retrieval simplified - use existing node > properties > > > and attributes instead of Compute > > > > > > On Tue, Apr 28, 2020 at 7:57 PM Igor Sapego > wrote: > > > > > >> I guess it makes sense. If anyone needs more control over connection > > >> we would need to implement a new feature anyway (like node filter we > > >> discussed earlier) > > >> > > >> Best Regards, > > >> Igor > > >> > > >> > > >> On Tue, Apr 28, 2020 at 12:29 PM Pavel Tupitsyn > > > >> wrote: > > >> > > >> > > enable the capability if the best effort affinity is on > > >> > I agree, makes sense. > > >> > > > >> > Igor, what do you think? > > >> > > > >> > On Tue, Apr 28, 2020 at 8:25 AM Denis Magda > > wrote: > > >> > > > >> > > Pavel, > > >> > > > > >> > > That would be a tremendous improvement for the recently release > best > > >> > effort > > >> > > affinity feature. Without this capability, we force application > > >> > developers > > >> > > to reopen thin client connections every type a cluster is scaled > > out. > > >> I > > >> > > believe that once the folks start using the best effort affinity, > > >> we'll > > >> > be > > >> > > hearing more of a feature request for what you're proposing in > this > > >> > thread. > > >> > > So, thanks for taking care of this proactively! > > >> > > > > >> > > As for the public API changes, do we really need any extra flag? I > > >> would > > >> > > enable the capability if the best effort affinity is on. For me, > > it's > > >> > just > > >> > > a natural improvement of the latter and it sounds reasonable to > > reuse > > >> the > > >> > > best effort affinity's flag. > > >> > > > > >> > > - > > >> > > Denis > > >> > > > > >> > > > > >> > > On Mon, Apr 27, 2020 at 2:58 AM Pavel Tupitsyn < > > ptupit...@apache.org> > > >> > > wrote: > > >> > > > > >> > > > Igniters, > > >> > > > > > >> > > > I've prepared an IEP [1] and a POC [2] for Thin Client Discovery > > >> > feature. > > >> > > > Let's discuss it here. > > >> > > > > > >> > > > In particular, I'd like to address the following points: > > >> > > > > > >> > > > 1. Value: do you think this would be a good feature to have? > > >> > > > 2. Public API changes: is a boolean property enough? Should we > > have > > >> > > > something more complex, so users can plug in custom logic to > > filter > > >> > > and/or > > >> > > > translate IPs and host names? > > >> > > > 3. Server-side implementation details: should we use Compute, > Node > > >> > > > Attributes, or something else to retrieve client endpoints from > > all > > >> > nodes > > >> > > > in cluster? > > >> > > > > > >> > > > [1] > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-44%3A+Thin+client+cluster+discovery > > >> > > > [2] https://github.com/apache/ignite/pull/7744 > > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-12932 > > >> > > > > > >> > > > > >> > > > >> > > > > > >
Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]
Done. > 7 мая 2020 г., в 13:20, Denis Garus написал(а): > > Nikolay, > could we add the simple improvement [1] to 2.8.1 scope? > > 1. https://issues.apache.org/jira/browse/IGNITE-12983 > > чт, 30 апр. 2020 г. в 11:26, Alex Plehanov : > >> Nikolay, >> >> TC results: [1], [2] >> I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1 >> >> [1]: >> >> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7754%2Fhead=Latest=ignite-2.8.1 >> [2]: >> >> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7755%2Fhead=Latest=ignite-2.8.1 >> >> вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov : >> >>> Hello, Alex. >>> >>> +1 from me. >>> 28 апр. 2020 г., в 15:03, Alex Plehanov >>> написал(а): Hello guys, While we are still waiting for some tickets to resolve I propose to cherry-pick to 2.8.1 two more bugfixes: IGNITE-12933 Fixed node failure after put incorrect key class for cache with indexed types IGNITE-12855 Fixed node failure with concurrent get operation and entry expiration when persistent is enabled Both fixes prevent node failure in some circumstances, both fixed >> already merged to master. WDYT? пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov : > Taras. > > Thank you, very much! > You changes merged to 2.8.1 branch. > > Igniters, > > We have 10 tickets scheduled for 2.8.1 release: > > OPEN: > > IGNITE-11687Concurrent WAL replay & log may fail with CRC error on > read - Dmitriy Govorukhin > IGNITE-12346.NET: Platform error:System.NullReferenceException - > Pavel Tupitsyn > > IN PROGRESS: > > IGNITE-12637IgniteSparkSession doesn't start the clients on really > distributed cluster - Yaroslav Molochkov > IGNITE-12788Cluster achieved fully rebalanced (PME-free ready) >> state > metric - Nikolay Izhikov > > PATCH AVAILABLE: > > IGNITE-10417notifyDiscoveryListener() call can be lost. - Pavel > Voronkin > IGNITE-12852Comma in field is not supported by COPY command - >> YuJue >>> Li > IGNITE-12252Unchecked exceptions during rebalancing should be >>> handled > - Nikolai Kulagin > IGNITE-12905QueryKeyValueIterable missing custom spliterator() > implementation - Johnny Galatikitis > IGNITE-12801Possible extra page release when throttling and >>> checkpoint > thread store its concurrently - Anton Kalashnikov > IGNITE-12794Scan query fails with an assertion error: Unexpected >> row > key - Denis Mekhanikov > > >> 27 апр. 2020 г., в 11:08, Taras Ledkov > написал(а): >> >> Hi, >> >> Nikolay, i've created PR [1] that contains the SQL-related tickets to > port into 2.8.1: >> >> IGNITE-12790 Introduce distributed SQL configuration and ability to > disable SQL functions. >> IGNITE-12887 Fix handle type mismatch exception on compare values >> while > traversing index tree. >> IGNITE-12848 fix H2Connection leaks on INSERT >> IGNITE-12800 SQL: local queries cursors must be closed or full read >> to > unlock the GridH2Table. >> >> TC test are OK. Please take a look at the TC bot report [2]. >> Please review & merge the patch into ignite-2.8.1. >> >> [1]. https://github.com/apache/ignite/pull/7703 >> [2]. > >>> >> https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7703%2Fhead=Latest=ignite-2.8.1 >> >> >> On 23.04.2020 13:25, Mikhail Petrov wrote: >>> Hello, Igniters. >>> >>> I propose to cherry-pick to 2.8.1 ticket [1]. >>> >>> >>> In addition to adding a new metric, it fixes a bug when, after > deactivation, GridDhtPartitionsExchangeFuture#rebalanced flag was not > reset. And therefore, it can be different on nodes that are already in >>> the > cluster from newly joined ones. >>> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-12788 >>> >>> Regards, >>> Mikhail. >>> >>> On 22.04.2020 14:03, Nikolay Izhikov wrote: Hello, Ivan. I think we can include this improvements. Please, go ahead. > 22 апр. 2020 г., в 10:21, Ivan Bessonov > написал(а): > > Hi Igniters, > > I'm continuing with IGNITE-12756 PR creation. It turned out that >> we > need 3 > more cherry-picks > to avoid massive code changes. Tests started failing after my >>> initial > attempt to create that PR. > > So, in total I have PR with 6 commits in it. Some of them fix > components > and tests for them, > others are required for new tests to compile and run without >> changes > in > 2.8.1 branch. > RunAll is in progress
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Maxim, congratulations! чт, 7 мая 2020 г. в 15:12, Denis Garus : > Maxim, Congrats! > Great job! > > чт, 7 мая 2020 г. в 15:02, Nikita Amelchev : > > > Maxim, congrats! > > > > чт, 7 мая 2020 г. в 14:55, Nikolay Izhikov : > > > > > > Congrats. > > > > > > > 7 мая 2020 г., в 14:54, Ivan Pavlukhin > > написал(а): > > > > > > > > Maxim, > > > > > > > > My congratulations! Well deserved! > > > > > > > > P.S. We should mention snapshots for persistent caches in the > > > > achievement list. Great job! > > > > > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > > > >> > > > >> The Project Management Committee (PMC) for Apache Ignite > > > >> > > > >> has invited Maxim Muzafarov to become new PMC member and we are > > pleased to > > > >> announce that he has accepted. > > > >> > > > >> > > > >> > > > >> Maxim is active dev list participant, speaker at meetups, and > > contributes > > > >> to additional checks of the product using travis and started new > file > > > >> rebalance. Maxim did a fantastic job to make release 2.8 possible > > > >> > > > >> > > > >> > > > >> Being a PMC member enables assistance with the management > > > >> > > > >> and to guide the direction of the project. > > > >> > > > >> Maxim, congrats with new role and keep the pace ! > > > >> > > > >> > > > >> > > > >> Best Regards, > > > >> > > > >> Dmitriy Pavlov > > > >> > > > >> on behalf of Apache Ignite PMC > > > > > > > > > -- > > Best wishes, > > Amelchev Nikita > > >
Re: Extended logging for rebalance performance analysis
Hi, that's it. 06.05.2020, 19:50, "Ivan Rakov" : > Hi, > > IGNITE_WRITE_REBALANCE_PARTITION_DISTRIBUTION_THRESHOLD - threshold >> duration rebalance of cache group after which partitions distribution is >> output, set in milliseconds, default value is 10 minutes. > > Does it mean that if the rebalancing process took less than 10 minutes, > only a short version of the message (with supplier statistics) will show up? > > In general, I have no objections. > > On Mon, May 4, 2020 at 10:38 AM ткаленко кирилл > wrote: > >> Hi, Igniters! >> >> I'd like to share a new small feature in AI [1]. >> >> Current rebalance logging does not allow you to quickly answer following >> questions: >> 1)How long was the balance(divided by supplier)? >> 2)How many records and bytes per supplier were rebalanced? >> 3)How many times did rebalance restart? >> 4)Which partitions were rebalanced and from which nodes did they receive >> them? >> 5)When did rebalance for all cache groups end? >> >> What you can see in logs now: >> >> 1)Starting rebalance with order of cache groups. >> Rebalancing scheduled [order=[ignite-sys-cache, grp1, grp0], >> top=AffinityTopologyVersion [topVer=2, minorTopVer=0], force=false, >> evt=NODE_JOINED, node=c2146a04-dc23-4bc9-870d-dfbb55c1] >> >> 2)Start rebalance of cache group from a specific supplier, specifying >> partition ids and mode - historical or full. >> Starting rebalance routine [ignite-sys-cache, >> topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], >> supplier=8c525892-703b-4fc4-b28b-b2f13970, fullPartitions=[0-99], >> histPartitions=[]] >> >> 3)Getting partial or complete partitions of cache group. >> Completed rebalancing [grp=ignite-sys-cache, >> supplier=8c525892-703b-4fc4-b28b-b2f13970, >> topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], progress=1/2] >> Completed (final) rebalancing [grp=ignite-sys-cache, >> supplier=c2146a04-dc23-4bc9-870d-dfbb55c1, >> topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], progress=2/2] >> >> 4)End rebalance of cache group. >> Completed rebalance future: RebalanceFuture [grp=CacheGroupContext >> [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=2, >> minorTopVer=0], rebalanceId=1, routines=1, receivedBytes=1200, >> receivedKeys=0, partitionsLeft=0, startTime=1588519707607, endTime=-1, >> lastCancelledTime=-1] >> >> Rebalance statistics: >> >> To speed up rebalance analysis, statistics will be output for each cache >> group and total for all cache groups. >> If duration rebalance for cache group is greater than threshold value, >> partition distribution is output. >> Statistics will you to analyze duration of the balance for each supplier >> to understand which of them has been transmitting data for longest time. >> >> System properties are used to output statistics: >> >> IGNITE_QUIET - to output statistics, value must be false; >> IGNITE_WRITE_REBALANCE_PARTITION_DISTRIBUTION_THRESHOLD - threshold >> duration rebalance of cache group after which partitions distribution is >> output, set in milliseconds, default value is 10 minutes. >> >> Statistics examples: >> >> Successful full and historical rebalance of group cache, without >> partitions distribution. >> Rebalance information per cache group (successful rebalance): [id=3181548, >> name=grp1, startTime=2020-04-13 10:55:16,117, finishTime=2020-04-13 >> 10:55:16,127, d=10 ms, restarted=0] Supplier statistics: [nodeId=0, p=5, >> d=10 ms] [nodeId=1, p=5, d=10 ms] Aliases: p - partitions, e - entries, b - >> bytes, d - duration, h - historical, nodeId mapping >> (nodeId=id,consistentId) [0=rebalancing.RebalanceStatisticsTest1] >> [1=rebalancing.RebalanceStatisticsTest0] >> Rebalance information per cache group (successful rebalance): [id=3181547, >> name=grp0, startTime=2020-04-13 15:01:44,000, finishTime=2020-04-13 >> 15:01:44,116, d=116 ms, restarted=0] Supplier statistics: [nodeId=0, hp=10, >> he=300, hb=30267, d=116 ms] Aliases: p - partitions, e - entries, b - >> bytes, d - duration, h - historical, nodeId mapping >> (nodeId=id,consistentId) [0=rebalancing.RebalanceStatisticsTest0] >> >> Successful full and historical rebalance of group cache, with partitions >> distribution. >> Rebalance information per cache group (successful rebalance): [id=3181548, >> name=grp1, startTime=2020-04-13 10:55:16,117, finishTime=2020-04-13 >> 10:55:16,127, d=10 ms, restarted=0] Supplier statistics: [nodeId=0, p=5, >> d=10 ms] [nodeId=1, p=5, d=10 ms] Aliases: p - partitions, e - entries, b - >> bytes, d - duration, h - historical, nodeId mapping >> (nodeId=id,consistentId) [0=rebalancing.RebalanceStatisticsTest1] >> [1=rebalancing.RebalanceStatisticsTest0] Rebalance duration was greater >> than 5 ms, printing detailed information about partitions distribution >> (threshold can be changed by setting number of milliseconds into >> IGNITE_WRITE_REBALANCE_PARTITION_DISTRIBUTION_THRESHOLD) 0 =
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Maxim, Congrats! Great job! чт, 7 мая 2020 г. в 15:02, Nikita Amelchev : > Maxim, congrats! > > чт, 7 мая 2020 г. в 14:55, Nikolay Izhikov : > > > > Congrats. > > > > > 7 мая 2020 г., в 14:54, Ivan Pavlukhin > написал(а): > > > > > > Maxim, > > > > > > My congratulations! Well deserved! > > > > > > P.S. We should mention snapshots for persistent caches in the > > > achievement list. Great job! > > > > > > Best regards, > > > Ivan Pavlukhin > > > > > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > > >> > > >> The Project Management Committee (PMC) for Apache Ignite > > >> > > >> has invited Maxim Muzafarov to become new PMC member and we are > pleased to > > >> announce that he has accepted. > > >> > > >> > > >> > > >> Maxim is active dev list participant, speaker at meetups, and > contributes > > >> to additional checks of the product using travis and started new file > > >> rebalance. Maxim did a fantastic job to make release 2.8 possible > > >> > > >> > > >> > > >> Being a PMC member enables assistance with the management > > >> > > >> and to guide the direction of the project. > > >> > > >> Maxim, congrats with new role and keep the pace ! > > >> > > >> > > >> > > >> Best Regards, > > >> > > >> Dmitriy Pavlov > > >> > > >> on behalf of Apache Ignite PMC > > > > > -- > Best wishes, > Amelchev Nikita >
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Maxim, congrats! чт, 7 мая 2020 г. в 14:55, Nikolay Izhikov : > > Congrats. > > > 7 мая 2020 г., в 14:54, Ivan Pavlukhin написал(а): > > > > Maxim, > > > > My congratulations! Well deserved! > > > > P.S. We should mention snapshots for persistent caches in the > > achievement list. Great job! > > > > Best regards, > > Ivan Pavlukhin > > > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > >> > >> The Project Management Committee (PMC) for Apache Ignite > >> > >> has invited Maxim Muzafarov to become new PMC member and we are pleased to > >> announce that he has accepted. > >> > >> > >> > >> Maxim is active dev list participant, speaker at meetups, and contributes > >> to additional checks of the product using travis and started new file > >> rebalance. Maxim did a fantastic job to make release 2.8 possible > >> > >> > >> > >> Being a PMC member enables assistance with the management > >> > >> and to guide the direction of the project. > >> > >> Maxim, congrats with new role and keep the pace ! > >> > >> > >> > >> Best Regards, > >> > >> Dmitriy Pavlov > >> > >> on behalf of Apache Ignite PMC > -- Best wishes, Amelchev Nikita
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Congrats. > 7 мая 2020 г., в 14:54, Ivan Pavlukhin написал(а): > > Maxim, > > My congratulations! Well deserved! > > P.S. We should mention snapshots for persistent caches in the > achievement list. Great job! > > Best regards, > Ivan Pavlukhin > > чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : >> >> The Project Management Committee (PMC) for Apache Ignite >> >> has invited Maxim Muzafarov to become new PMC member and we are pleased to >> announce that he has accepted. >> >> >> >> Maxim is active dev list participant, speaker at meetups, and contributes >> to additional checks of the product using travis and started new file >> rebalance. Maxim did a fantastic job to make release 2.8 possible >> >> >> >> Being a PMC member enables assistance with the management >> >> and to guide the direction of the project. >> >> Maxim, congrats with new role and keep the pace ! >> >> >> >> Best Regards, >> >> Dmitriy Pavlov >> >> on behalf of Apache Ignite PMC
Re: [ANNOUNCE] New PMC member: Maxim Muzafarov
Maxim, My congratulations! Well deserved! P.S. We should mention snapshots for persistent caches in the achievement list. Great job! Best regards, Ivan Pavlukhin чт, 7 мая 2020 г. в 14:47, Dmitriy Pavlov : > > The Project Management Committee (PMC) for Apache Ignite > > has invited Maxim Muzafarov to become new PMC member and we are pleased to > announce that he has accepted. > > > > Maxim is active dev list participant, speaker at meetups, and contributes > to additional checks of the product using travis and started new file > rebalance. Maxim did a fantastic job to make release 2.8 possible > > > > Being a PMC member enables assistance with the management > > and to guide the direction of the project. > > Maxim, congrats with new role and keep the pace ! > > > > Best Regards, > > Dmitriy Pavlov > > on behalf of Apache Ignite PMC
Re: [DISCUSSION] Ignite WebConsole Deprecation
Hi Igniters, Only whole project may go to the attic. Main reason of moving project to the attic is inactive community and less than 3 PMC members voting for new release. So attic is more about waiting for new community to resurrect project, and less about codebase. I suggest us, as always, create new repo and state that development of the module was discontinued, for now. Additionally, discontinuation is essential and really major community-wide decision, it requires formal vote to be run after this discussion. I'm sure if we build consensus in advance, here in the discussion, vote would be just formality, but it is still needed. Sincerely, Dmitriy Pavlov чт, 7 мая 2020 г. в 12:34, Ilya Kasnacheev : > Hello! > > I think we should move it to a separate repository, but not Attic. > > I can see the scenario where there would be recurrence of interest to this > tool and somebody may step in to update it. > > Regards, > -- > Ilya Kasnacheev > > > ср, 6 мая 2020 г. в 22:20, Denis Magda : > > > Folks, > > > > Personally, I like the idea of Apache Attic [1] suggested by Saikat. We > are > > not discontinuing Ignite WebConsole because it became useless like it was > > with Hadoop Accelerator. The tool has good capabilities such as the > > configuration wizard, and the only problem is that we no longer want to > > support or maintain this technology (while it has the right to exist). > > > > Is anybody interested in figuring out with the ASF-mates if Ignite > > WebConsole can be accepted to Attic? The Attic's guidelines talk about an > > ASF project as a whole and not about individual components. > > > > [1] https://attic.apache.org > > > > - > > Denis > > > > > > On Wed, May 6, 2020 at 7:20 AM Вячеслав Коптилин < > slava.kopti...@gmail.com > > > > > wrote: > > > > > Hello, > > > > > > +1 to remove this component or move it to a separate repository if > > someone > > > wants to maintain it. > > > In case the web console provides useful features, we should consider > how > > to > > > add them to our command-line utilities, if possible. > > > > > > Thanks, > > > Slava. > > > > > > ср, 6 мая 2020 г. в 16:10, Nikolay Izhikov : > > > > > > > Hello. > > > > > > > > +1 to remove any graphical utilities from the Ignite core. > > > > I think we should maintain and support only cmd, JMX, and other > «core» > > > > tools. > > > > > > > > We shouldn't waste our resources on supporting pretty looking UI > tools. > > > > > > > > > > > > > 2 мая 2020 г., в 04:05, Saikat Maitra > > > > написал(а): > > > > > > > > > > Hello Denis, > > > > > > > > > > I am thinking if we should move web-console to a separate repo like > > > > > ignite-web-console. In my opinion the tech stack we have in > > web-console > > > > > like npm, nodejs and Mondodb etc are little different and can be > > hosted > > > > as > > > > > a separate project in a git repo. > > > > > > > > > > I had used web-console for streamers project and found it helpful > in > > > > > creating table schema and execute sql queries and also to do cache > > > > lookup. > > > > > > > > > > https://github.com/samaitra/streamers > > > > > > > > > > If we continue to find that usage of ignite-web-console is limited > > then > > > > we > > > > > can plan moving ignite-web-console in Apache Attic > > > > > > > > > > https://attic.apache.org/ > > > > > > > > > > Please let me know your thoughts. > > > > > > > > > > Regards, > > > > > Saikat > > > > > > > > > > > > > > > > > > > > On Fri, May 1, 2020 at 12:43 PM Denis Magda > > wrote: > > > > > > > > > >> Igniters, > > > > >> > > > > >> I would like to hear your opinion on what we should do with Ignite > > > > >> WebConsole. > > > > >> > > > > >> To my knowledge, we don't have active maintainers of the > component, > > > and > > > > a > > > > >> list of issues is piling up [1]. Users even report that the docker > > > > images > > > > >> have not been updated for more than a year [2]. > > > > >> > > > > >> Personally, I share the opinion of those who believe the community > > > > needs to > > > > >> provide and support all the essential tooling (metrics and tracing > > > API, > > > > >> command-line tool) while the UI tools are not our business. There > > is a > > > > >> myriad of UI tools Ignite can be monitored and traced with. Users > > > > already > > > > >> have plenty of choices. > > > > >> > > > > >> What are your thoughts? Probably, some of you want to become > > > > maintainers of > > > > >> the component. Otherwise, we should let the tool go. > > > > >> > > > > >> [1] > > > > >> > > > > >> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12923?jql=project%20%3D%20IGNITE%20AND%20text%20~%20%22web%20console%22%20AND%20status%20%3D%20Open%20and%20type%20%3D%20Bug%20ORDER%20BY%20createdDate%20 > > > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12923 > > > > >> > > > > >> - > > > > >> Denis > > > > >> > > > > > > > > > > > > > >
[ANNOUNCE] New PMC member: Maxim Muzafarov
The Project Management Committee (PMC) for Apache Ignite has invited Maxim Muzafarov to become new PMC member and we are pleased to announce that he has accepted. Maxim is active dev list participant, speaker at meetups, and contributes to additional checks of the product using travis and started new file rebalance. Maxim did a fantastic job to make release 2.8 possible Being a PMC member enables assistance with the management and to guide the direction of the project. Maxim, congrats with new role and keep the pace ! Best Regards, Dmitriy Pavlov on behalf of Apache Ignite PMC
[jira] [Created] (IGNITE-12990) Remove static thread group from IgniteSpiThread
Alexey Goncharuk created IGNITE-12990: - Summary: Remove static thread group from IgniteSpiThread Key: IGNITE-12990 URL: https://issues.apache.org/jira/browse/IGNITE-12990 Project: Ignite Issue Type: Improvement Reporter: Alexey Goncharuk This is a follow-up ticket for IGNITE-12554. The static thread group has not been removed from IgniteSpiThread which still prevents node start after the thread group destroy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12989) Tx recovery should log statuses to TimeBag
Anton Vinogradov created IGNITE-12989: - Summary: Tx recovery should log statuses to TimeBag Key: IGNITE-12989 URL: https://issues.apache.org/jira/browse/IGNITE-12989 Project: Ignite Issue Type: Task Reporter: Anton Vinogradov Assignee: Anton Vinogradov Fix For: 2.9 Timings, counters and other important info should be logged via TimeBag. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12988) c++ thin client installer misses some headers
Bastien Durel created IGNITE-12988:
--
Summary: c++ thin client installer misses some headers
Key: IGNITE-12988
URL: https://issues.apache.org/jira/browse/IGNITE-12988
Project: Ignite
Issue Type: Bug
Components: build, examples, thin client
Affects Versions: 2.8
Reporter: Bastien Durel
Hello,
After building and installing (with make install) the c++ module for ignite,
some header files are missing, so the most simple example cannot compile :
{code:java}
make: Entering directory '/home/bastien/project/webix/cpp'
g++ -I/usr/local/include -c -o data.o data.cpp
In file included from
/usr/local/include/ignite/impl/binary/binary_writer_impl.h:32,
from /usr/local/include/ignite/binary/binary_raw_writer.h:30,
from /usr/local/include/ignite/impl/thin/writable.h:21,
from /usr/local/include/ignite/thin/cache/cache_client.h:28,
from /usr/local/include/ignite/thin/ignite_client.h:31,
from data.cpp:3:
/usr/local/include/ignite/impl/binary/binary_utils.h:30:10: fatal error:
ignite/binary/binary_enum_entry.h: No such file or directory
30 | #include
| ^~~
compilation terminated.
make: *** [: data.o] Error 1
{code}
To make the first thin client example compile, I had to manually copy
`binary_enum_entry.h` and `binary_enum.h` to /usr/local/include/ignite/binary/
Using the deb file from [http://apache.org/dist/ignite/deb/] leads to the same
results.
While trying to compile a non-thin client, I ran into a similar issue with
which is not installed, but I did not push this way.
By the way, the examples on
[https://apacheignite-cpp.readme.io/docs/thin-client] reffers to a non-existant
include file :
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]
Nikolay, could we add the simple improvement [1] to 2.8.1 scope? 1. https://issues.apache.org/jira/browse/IGNITE-12983 чт, 30 апр. 2020 г. в 11:26, Alex Plehanov : > Nikolay, > > TC results: [1], [2] > I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1 > > [1]: > > https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7754%2Fhead=Latest=ignite-2.8.1 > [2]: > > https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7755%2Fhead=Latest=ignite-2.8.1 > > вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov : > > > Hello, Alex. > > > > +1 from me. > > > > > 28 апр. 2020 г., в 15:03, Alex Plehanov > > написал(а): > > > > > > Hello guys, > > > > > > While we are still waiting for some tickets to resolve I propose to > > > cherry-pick to 2.8.1 two more bugfixes: > > > IGNITE-12933 Fixed node failure after put incorrect key class for cache > > > with indexed types > > > IGNITE-12855 Fixed node failure with concurrent get operation and entry > > > expiration when persistent is enabled > > > Both fixes prevent node failure in some circumstances, both fixed > already > > > merged to master. > > > > > > WDYT? > > > > > > пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov : > > > > > >> Taras. > > >> > > >> Thank you, very much! > > >> You changes merged to 2.8.1 branch. > > >> > > >> Igniters, > > >> > > >> We have 10 tickets scheduled for 2.8.1 release: > > >> > > >> OPEN: > > >> > > >> IGNITE-11687Concurrent WAL replay & log may fail with CRC error on > > >> read - Dmitriy Govorukhin > > >> IGNITE-12346.NET: Platform error:System.NullReferenceException - > > >> Pavel Tupitsyn > > >> > > >> IN PROGRESS: > > >> > > >> IGNITE-12637IgniteSparkSession doesn't start the clients on really > > >> distributed cluster - Yaroslav Molochkov > > >> IGNITE-12788Cluster achieved fully rebalanced (PME-free ready) > state > > >> metric - Nikolay Izhikov > > >> > > >> PATCH AVAILABLE: > > >> > > >> IGNITE-10417notifyDiscoveryListener() call can be lost. - Pavel > > >> Voronkin > > >> IGNITE-12852Comma in field is not supported by COPY command - > YuJue > > Li > > >> IGNITE-12252Unchecked exceptions during rebalancing should be > > handled > > >> - Nikolai Kulagin > > >> IGNITE-12905QueryKeyValueIterable missing custom spliterator() > > >> implementation - Johnny Galatikitis > > >> IGNITE-12801Possible extra page release when throttling and > > checkpoint > > >> thread store its concurrently - Anton Kalashnikov > > >> IGNITE-12794Scan query fails with an assertion error: Unexpected > row > > >> key - Denis Mekhanikov > > >> > > >> > > >>> 27 апр. 2020 г., в 11:08, Taras Ledkov > > >> написал(а): > > >>> > > >>> Hi, > > >>> > > >>> Nikolay, i've created PR [1] that contains the SQL-related tickets to > > >> port into 2.8.1: > > >>> > > >>> IGNITE-12790 Introduce distributed SQL configuration and ability to > > >> disable SQL functions. > > >>> IGNITE-12887 Fix handle type mismatch exception on compare values > while > > >> traversing index tree. > > >>> IGNITE-12848 fix H2Connection leaks on INSERT > > >>> IGNITE-12800 SQL: local queries cursors must be closed or full read > to > > >> unlock the GridH2Table. > > >>> > > >>> TC test are OK. Please take a look at the TC bot report [2]. > > >>> Please review & merge the patch into ignite-2.8.1. > > >>> > > >>> [1]. https://github.com/apache/ignite/pull/7703 > > >>> [2]. > > >> > > > https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7703%2Fhead=Latest=ignite-2.8.1 > > >>> > > >>> > > >>> On 23.04.2020 13:25, Mikhail Petrov wrote: > > Hello, Igniters. > > > > I propose to cherry-pick to 2.8.1 ticket [1]. > > > > > > In addition to adding a new metric, it fixes a bug when, after > > >> deactivation, GridDhtPartitionsExchangeFuture#rebalanced flag was not > > >> reset. And therefore, it can be different on nodes that are already in > > the > > >> cluster from newly joined ones. > > > > [1] - https://issues.apache.org/jira/browse/IGNITE-12788 > > > > Regards, > > Mikhail. > > > > On 22.04.2020 14:03, Nikolay Izhikov wrote: > > > Hello, Ivan. > > > > > > I think we can include this improvements. > > > Please, go ahead. > > > > > >> 22 апр. 2020 г., в 10:21, Ivan Bessonov > > >> написал(а): > > >> > > >> Hi Igniters, > > >> > > >> I'm continuing with IGNITE-12756 PR creation. It turned out that > we > > >> need 3 > > >> more cherry-picks > > >> to avoid massive code changes. Tests started failing after my > > initial > > >> attempt to create that PR. > > >> > > >> So, in total I have PR with 6 commits in it. Some of them fix > > >> components > > >> and tests for them, > > >> others are required for new tests to compile and run without > changes > > >> in > > >> 2.8.1 branch. > > >> RunAll is in progress now and I'll reply with
Re: [DISCUSSION] Ignite WebConsole Deprecation
Hello! I think we should move it to a separate repository, but not Attic. I can see the scenario where there would be recurrence of interest to this tool and somebody may step in to update it. Regards, -- Ilya Kasnacheev ср, 6 мая 2020 г. в 22:20, Denis Magda : > Folks, > > Personally, I like the idea of Apache Attic [1] suggested by Saikat. We are > not discontinuing Ignite WebConsole because it became useless like it was > with Hadoop Accelerator. The tool has good capabilities such as the > configuration wizard, and the only problem is that we no longer want to > support or maintain this technology (while it has the right to exist). > > Is anybody interested in figuring out with the ASF-mates if Ignite > WebConsole can be accepted to Attic? The Attic's guidelines talk about an > ASF project as a whole and not about individual components. > > [1] https://attic.apache.org > > - > Denis > > > On Wed, May 6, 2020 at 7:20 AM Вячеслав Коптилин > > wrote: > > > Hello, > > > > +1 to remove this component or move it to a separate repository if > someone > > wants to maintain it. > > In case the web console provides useful features, we should consider how > to > > add them to our command-line utilities, if possible. > > > > Thanks, > > Slava. > > > > ср, 6 мая 2020 г. в 16:10, Nikolay Izhikov : > > > > > Hello. > > > > > > +1 to remove any graphical utilities from the Ignite core. > > > I think we should maintain and support only cmd, JMX, and other «core» > > > tools. > > > > > > We shouldn't waste our resources on supporting pretty looking UI tools. > > > > > > > > > > 2 мая 2020 г., в 04:05, Saikat Maitra > > > написал(а): > > > > > > > > Hello Denis, > > > > > > > > I am thinking if we should move web-console to a separate repo like > > > > ignite-web-console. In my opinion the tech stack we have in > web-console > > > > like npm, nodejs and Mondodb etc are little different and can be > hosted > > > as > > > > a separate project in a git repo. > > > > > > > > I had used web-console for streamers project and found it helpful in > > > > creating table schema and execute sql queries and also to do cache > > > lookup. > > > > > > > > https://github.com/samaitra/streamers > > > > > > > > If we continue to find that usage of ignite-web-console is limited > then > > > we > > > > can plan moving ignite-web-console in Apache Attic > > > > > > > > https://attic.apache.org/ > > > > > > > > Please let me know your thoughts. > > > > > > > > Regards, > > > > Saikat > > > > > > > > > > > > > > > > On Fri, May 1, 2020 at 12:43 PM Denis Magda > wrote: > > > > > > > >> Igniters, > > > >> > > > >> I would like to hear your opinion on what we should do with Ignite > > > >> WebConsole. > > > >> > > > >> To my knowledge, we don't have active maintainers of the component, > > and > > > a > > > >> list of issues is piling up [1]. Users even report that the docker > > > images > > > >> have not been updated for more than a year [2]. > > > >> > > > >> Personally, I share the opinion of those who believe the community > > > needs to > > > >> provide and support all the essential tooling (metrics and tracing > > API, > > > >> command-line tool) while the UI tools are not our business. There > is a > > > >> myriad of UI tools Ignite can be monitored and traced with. Users > > > already > > > >> have plenty of choices. > > > >> > > > >> What are your thoughts? Probably, some of you want to become > > > maintainers of > > > >> the component. Otherwise, we should let the tool go. > > > >> > > > >> [1] > > > >> > > > >> > > > > > > https://issues.apache.org/jira/browse/IGNITE-12923?jql=project%20%3D%20IGNITE%20AND%20text%20~%20%22web%20console%22%20AND%20status%20%3D%20Open%20and%20type%20%3D%20Bug%20ORDER%20BY%20createdDate%20 > > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12923 > > > >> > > > >> - > > > >> Denis > > > >> > > > > > > > > >
Re: IEP-44 Thin Client Discovery
Hello! I think it is OK if client loses cluster, in this case it should only check the addresses with which it was configured to start. Regards, -- Ilya Kasnacheev ср, 6 мая 2020 г. в 17:31, Pavel Tupitsyn : > Igniters, let's discuss the following issue: > > For partition awareness, and now for cluster discovery, we use a response > flag to detect topology changes. > The problem is - if the client does not do anything (user code does not > perform operations), > then we'll never know about topology changes and may even lose the cluster > (all known nodes leave). > > Should we introduce some keep-alive mechanism, so that thin clients send > periodic ping requests? > Maybe do this as a separate feature. > > Thoughts? > > On Tue, Apr 28, 2020 at 8:14 PM Pavel Tupitsyn > wrote: > > > Ok, I've updated IEP and POC accordingly: > > * Config flag removed > > * IPs and host names retrieval simplified - use existing node properties > > and attributes instead of Compute > > > > On Tue, Apr 28, 2020 at 7:57 PM Igor Sapego wrote: > > > >> I guess it makes sense. If anyone needs more control over connection > >> we would need to implement a new feature anyway (like node filter we > >> discussed earlier) > >> > >> Best Regards, > >> Igor > >> > >> > >> On Tue, Apr 28, 2020 at 12:29 PM Pavel Tupitsyn > >> wrote: > >> > >> > > enable the capability if the best effort affinity is on > >> > I agree, makes sense. > >> > > >> > Igor, what do you think? > >> > > >> > On Tue, Apr 28, 2020 at 8:25 AM Denis Magda > wrote: > >> > > >> > > Pavel, > >> > > > >> > > That would be a tremendous improvement for the recently release best > >> > effort > >> > > affinity feature. Without this capability, we force application > >> > developers > >> > > to reopen thin client connections every type a cluster is scaled > out. > >> I > >> > > believe that once the folks start using the best effort affinity, > >> we'll > >> > be > >> > > hearing more of a feature request for what you're proposing in this > >> > thread. > >> > > So, thanks for taking care of this proactively! > >> > > > >> > > As for the public API changes, do we really need any extra flag? I > >> would > >> > > enable the capability if the best effort affinity is on. For me, > it's > >> > just > >> > > a natural improvement of the latter and it sounds reasonable to > reuse > >> the > >> > > best effort affinity's flag. > >> > > > >> > > - > >> > > Denis > >> > > > >> > > > >> > > On Mon, Apr 27, 2020 at 2:58 AM Pavel Tupitsyn < > ptupit...@apache.org> > >> > > wrote: > >> > > > >> > > > Igniters, > >> > > > > >> > > > I've prepared an IEP [1] and a POC [2] for Thin Client Discovery > >> > feature. > >> > > > Let's discuss it here. > >> > > > > >> > > > In particular, I'd like to address the following points: > >> > > > > >> > > > 1. Value: do you think this would be a good feature to have? > >> > > > 2. Public API changes: is a boolean property enough? Should we > have > >> > > > something more complex, so users can plug in custom logic to > filter > >> > > and/or > >> > > > translate IPs and host names? > >> > > > 3. Server-side implementation details: should we use Compute, Node > >> > > > Attributes, or something else to retrieve client endpoints from > all > >> > nodes > >> > > > in cluster? > >> > > > > >> > > > [1] > >> > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-44%3A+Thin+client+cluster+discovery > >> > > > [2] https://github.com/apache/ignite/pull/7744 > >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-12932 > >> > > > > >> > > > >> > > >> > > >
Re: [DISCUSS] Data loss handling improvements
Yes, it will work this way. чт, 7 мая 2020 г. в 10:43, Anton Vinogradov : > Seems I got the vision, thanks. > There should be only 2 ways to reset lost partition: to gain an owner from > resurrected first or to remove ex-owner from baseline (partition will be > rearranged). > And we should make a decision for every lost partition before calling the > reset. > > On Wed, May 6, 2020 at 8:02 PM Alexei Scherbakov < > alexey.scherbak...@gmail.com> wrote: > > > ср, 6 мая 2020 г. в 12:54, Anton Vinogradov : > > > > > Alexei, > > > > > > 1,2,4,5 - looks good to me, no objections here. > > > > > > >> 3. Lost state is impossible to reset if a topology doesn't have at > > least > > > >> one owner for each lost partition. > > > > > > Do you mean that, according to your example, where > > > >> a node2 has left, soon a node3 has left. If the node2 is returned to > > > >> the topology first, it would have stale data for some keys. > > > we have to have node2 at cluster to be able to reset "lost" to node2's > > > data? > > > > > > > Not sure if I understand a question, but try to answer using an example: > > Assume 3 nodes n1, n2, n3, 1 backup, persistence enabled, partition p is > > owned by n2 and n3. > > 1. Topology is activated. > > 2. cache.put(p, 0) // n2 and n3 have p->0, updateCounter=1 > > 3. n2 has failed. > > 4. cache.put(p, 1) // n3 has p->1, updateCounter=2 > > 5. n3 has failed, partition loss is happened. > > 6. n2 joins a topology, it has stale data (p->0) > > > > We actually have 2 issues: > > 7. cache.put(p, 2) will success, n2 has p->2, n3 has p->0, data is > diverged > > and will not be adjusted by counters rebalancing if n3 is later joins a > > topology. > > or > > 8. n3 joins a topology, it has actual data (p->1) but rebalancing will > not > > work because joining node has highest counter (it can only be a demander > in > > this scenario). > > > > In both cases rebalancing by counters will not work causing data > divergence > > in copies. > > > > > > > > > > >> at least one owner for each lost partition. > > > What the reason to have owners for all lost partitions when we want to > > > reset only some (available)? > > > > > > > It's never were possible to reset only subset of lost partitions. The > > reason is to make guarantee of resetLostPartitions method - all cache > > operations are resumed, data is correct. > > > > > > > Will it be possible to perform operations on non-lost partitions when > the > > > cluster has at least one lost partition? > > > > > > > Yes it will be. > > > > > > > > > > On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov < > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > Folks, > > > > > > > > I've almost finished a patch bringing some improvements to the data > > loss > > > > handling code, and I wish to discuss proposed changes with the > > community > > > > before submitting. > > > > > > > > *The issue* > > > > > > > > During the grid's lifetime, it's possible to get into a situation > when > > > some > > > > data nodes have failed or mistakenly stopped. If a number of stopped > > > nodes > > > > exceeds a certain threshold depending on configured backups, count a > > data > > > > loss will occur. For example, a grid having one backup (meaning at > > least > > > > two copies of each data partition exist at the same time) can > tolerate > > > only > > > > one node loss at the time. Generally, data loss is guaranteed to > occur > > if > > > > backups + 1 or more nodes have failed simultaneously using default > > > affinity > > > > function. > > > > > > > > For in-memory caches, data is lost forever. For persistent caches, > data > > > is > > > > not physically lost and accessible again after failed nodes are > > returned > > > to > > > > the topology. > > > > > > > > Possible data loss should be taken into consideration while designing > > an > > > > application. > > > > > > > > > > > > > > > > *Consider an example: money is transferred from one deposit to > another, > > > and > > > > all nodes holding data for one of the deposits are gone.In such a > > case, a > > > > transaction temporary cannot be completed until a cluster is > recovered > > > from > > > > the data loss state. Ignoring this can cause data inconsistency.* > > > > It is necessary to have an API telling us if an operation is safe to > > > > complete from the perspective of data loss. > > > > > > > > Such an API exists for some time [1] [2] [3]. In short, a grid can be > > > > configured to switch caches to the partial availability mode if data > > loss > > > > is detected. > > > > > > > > Let's give two definitions according to the Javadoc for > > > > *PartitionLossPolicy*: > > > > > > > > · *Safe* (data loss handling) *policy* - cache operations are only > > > > available for non-lost partitions (PartitionLossPolicy != IGNORE). > > > > > > > > · *Unsafe policy* - cache operations are always possible > > > > (PartitionLossPolicy = IGNORE). If the unsafe policy is configured, > > lost > > > >
Re: [DISCUSS] Data loss handling improvements
Seems I got the vision, thanks. There should be only 2 ways to reset lost partition: to gain an owner from resurrected first or to remove ex-owner from baseline (partition will be rearranged). And we should make a decision for every lost partition before calling the reset. On Wed, May 6, 2020 at 8:02 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > ср, 6 мая 2020 г. в 12:54, Anton Vinogradov : > > > Alexei, > > > > 1,2,4,5 - looks good to me, no objections here. > > > > >> 3. Lost state is impossible to reset if a topology doesn't have at > least > > >> one owner for each lost partition. > > > > Do you mean that, according to your example, where > > >> a node2 has left, soon a node3 has left. If the node2 is returned to > > >> the topology first, it would have stale data for some keys. > > we have to have node2 at cluster to be able to reset "lost" to node2's > > data? > > > > Not sure if I understand a question, but try to answer using an example: > Assume 3 nodes n1, n2, n3, 1 backup, persistence enabled, partition p is > owned by n2 and n3. > 1. Topology is activated. > 2. cache.put(p, 0) // n2 and n3 have p->0, updateCounter=1 > 3. n2 has failed. > 4. cache.put(p, 1) // n3 has p->1, updateCounter=2 > 5. n3 has failed, partition loss is happened. > 6. n2 joins a topology, it has stale data (p->0) > > We actually have 2 issues: > 7. cache.put(p, 2) will success, n2 has p->2, n3 has p->0, data is diverged > and will not be adjusted by counters rebalancing if n3 is later joins a > topology. > or > 8. n3 joins a topology, it has actual data (p->1) but rebalancing will not > work because joining node has highest counter (it can only be a demander in > this scenario). > > In both cases rebalancing by counters will not work causing data divergence > in copies. > > > > > > >> at least one owner for each lost partition. > > What the reason to have owners for all lost partitions when we want to > > reset only some (available)? > > > > It's never were possible to reset only subset of lost partitions. The > reason is to make guarantee of resetLostPartitions method - all cache > operations are resumed, data is correct. > > > > Will it be possible to perform operations on non-lost partitions when the > > cluster has at least one lost partition? > > > > Yes it will be. > > > > > > On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov < > > alexey.scherbak...@gmail.com> wrote: > > > > > Folks, > > > > > > I've almost finished a patch bringing some improvements to the data > loss > > > handling code, and I wish to discuss proposed changes with the > community > > > before submitting. > > > > > > *The issue* > > > > > > During the grid's lifetime, it's possible to get into a situation when > > some > > > data nodes have failed or mistakenly stopped. If a number of stopped > > nodes > > > exceeds a certain threshold depending on configured backups, count a > data > > > loss will occur. For example, a grid having one backup (meaning at > least > > > two copies of each data partition exist at the same time) can tolerate > > only > > > one node loss at the time. Generally, data loss is guaranteed to occur > if > > > backups + 1 or more nodes have failed simultaneously using default > > affinity > > > function. > > > > > > For in-memory caches, data is lost forever. For persistent caches, data > > is > > > not physically lost and accessible again after failed nodes are > returned > > to > > > the topology. > > > > > > Possible data loss should be taken into consideration while designing > an > > > application. > > > > > > > > > > > > *Consider an example: money is transferred from one deposit to another, > > and > > > all nodes holding data for one of the deposits are gone.In such a > case, a > > > transaction temporary cannot be completed until a cluster is recovered > > from > > > the data loss state. Ignoring this can cause data inconsistency.* > > > It is necessary to have an API telling us if an operation is safe to > > > complete from the perspective of data loss. > > > > > > Such an API exists for some time [1] [2] [3]. In short, a grid can be > > > configured to switch caches to the partial availability mode if data > loss > > > is detected. > > > > > > Let's give two definitions according to the Javadoc for > > > *PartitionLossPolicy*: > > > > > > · *Safe* (data loss handling) *policy* - cache operations are only > > > available for non-lost partitions (PartitionLossPolicy != IGNORE). > > > > > > · *Unsafe policy* - cache operations are always possible > > > (PartitionLossPolicy = IGNORE). If the unsafe policy is configured, > lost > > > partitions automatically re-created on the remaining nodes if needed or > > > immediately owned if a last supplier has left during rebalancing. > > > > > > *That needs to be fixed* > > > > > > 1. The default loss policy is unsafe, even for persistent caches in the > > > current implementation. It can result in unintentional data loss and > > > business invariants'
Re: [DISCUSS] Remove TC badge from README.md
Folks, I created a ticket [1] and prepared PR with magic number =). Please review. [1] https://issues.apache.org/jira/browse/IGNITE-12987 Best regards, Ivan Pavlukhin пн, 4 мая 2020 г. в 10:01, Ivan Pavlukhin : > > Hi Igniters, > > Inspired by a neighbor thread about PR checks [1]. It brought my > attention that now we have a neat Travis badge and a strange TC badge > which is not very useful from a first glance. > > What do you think, is it ought to completely remove TC badge from readme? > > [1] > https://lists.apache.org/thread.html/r1580e2fb23728e83afbe1bad2b4cf7e9cece0e19a1d305e5755d7dd1%40%3Cdev.ignite.apache.org%3E > > Best regards, > Ivan Pavlukhin
[jira] [Created] (IGNITE-12987) Remove TC badge from README.md
Ivan Pavlukhin created IGNITE-12987: --- Summary: Remove TC badge from README.md Key: IGNITE-12987 URL: https://issues.apache.org/jira/browse/IGNITE-12987 Project: Ignite Issue Type: Task Reporter: Ivan Pavlukhin Currently TC badge in main [README|https://github.com/apache/ignite/blob/master/README.md] seems to be not very useful because a build status is not seen directly (is says "no permissions to get data"). Moreover TC results are not very representative because integration tests are not stable by nature and we rely on TC bot checks instead of plain TC status. Let's remove this badge. -- This message was sent by Atlassian Jira (v8.3.4#803005)
