Re: [DISCUSSION] ignite-extension - ignite-spring-boot-autoconfigurer release.
Hi Nikolay, In fact I did not mean anything supernatural. A scheme you mentioned > I thought we just release module with the 1.0.0 version and specify supported > Ignite version somewhere in the documentation. sounds fine to me. Best regards, Ivan Pavlukhin чт, 6 февр. 2020 г. в 15:00, Nikolay Izhikov : > > Hello, Ivan. > > > As usual we need to decide about versioning and a correspondence to Ignite > > versions > > As we are going to have a separate release cycle I can imagine an > > independent versioning scheme with a range of supported Ignite versions. > > Please, clarify, what do you mean? > Do you have any idea how it should be implemented? > > I thought we just release module with the 1.0.0 version and specify supported > Ignite version somewhere in the documentation. > > > 6 февр. 2020 г., в 11:15, Ivan Pavlukhin написал(а): > > > > Nikolay, > > > > Thank you for driving it! It is great to establish this process in > > practice earlier because it seems that we need to release a Flink > > integration soon because Ignite 2.8 is going to be released without > > that integration bundled. > > > > As usual we need to decide about versioning and a correspondence to > > Ignite versions. As we are going to have a separate release cycle I > > can imagine an independent versioning scheme with a range of supported > > Ignite versions. > > > > Also I think it is a good idea to update ignite-extensions README on > > GitHub and provide a link to relevant TC jobs. > > > > Best regards, > > Ivan Pavlukhin > > > > ср, 5 февр. 2020 г. в 17:44, Denis Magda : > >> > >> Alex, did you have a chance to review Saikat’s changes related to the > >> extensions repository organization and release approach? > >> > >> Denis > >> > >> On Wednesday, February 5, 2020, Nikolay Izhikov > >> wrote: > >> > >>> Hello, Igniters. > >>> > >>> We don't have a release process for newly created ignite-extensions > >>> modules. > >>> I want to release two modules: > >>> > >>> * ignite-spring-boot-autoconfigure > >>> * ignite-client-spring-boot-autoconfigure > >>> > >>> Let's discuss it. > >>> > >>> Any objections to it? > >>> What should be done before release? > >>> > >> > >> > >> -- > >> - > >> Denis >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vladimir,
IGNITE_REUSE_MEMORY_ON_DEACTIVATE doesn't prevent cache contents from
clearing.
It allows allocated memory reuse on re-activation to avoid OS specific
issues if allocated heap is rather large.
You are right to create separate ticket for implementing required behavior.
чт, 6 февр. 2020 г. в 16:37, Vladimir Steshin :
> Or, is flag [1] is actual only for persistence mode? Related ticket [2] is
> completely about disabled persistence.
> If previous assumption is right, I think, we can involve flag [1] in ticket
> [2].
>
> [1]
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> [2] https://issues.apache.org/jira/browse/IGNITE-12614
>
> чт, 6 февр. 2020 г. в 16:10, Vladimir Steshin :
>
> > Denis, Alexei,
> >
> > Regarding usage of flag
> >
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> > [1]
> >
> > When enabled, I think the following test should work. But it fails.
> >
> > //
> > @Test
> > public void testDataPresent() throws Exception {
> > System.setProperty(IGNITE_REUSE_MEMORY_ON_DEACTIVATE, "true");
> >
> > IgniteEx i = startGrid(0);
> >
> > assertFalse(
> >
> i.configuration().getDataStorageConfiguration().getDefaultDataRegionConfiguration()
> > .isPersistenceEnabled() );
> >
> > String name = "non-persistent-cache";
> >
> > i.createCache(name).put(1L, 1L);
> >
> > assertEquals(1L, i.cache(name).get(1L));
> >
> > i.cluster().state(ClusterState.INACTIVE);
> > i.cluster().state(ClusterState.ACTIVE);
> >
> > assertEquals(1L, i.cache(name).get(1L)); //Assertion error here!
> > }
> > //
> >
> >
> > Several notes:
> >
> > - IgniteCacheDatabaseSharedManager#reuseMemory
> > is true
> > - IgniteCacheDatabaseSharedManager#onDeActivate(boolean shutdown)
> > is called with shutdown == false
> > - PageMemoryNoStoreImpl#stop(booleam deallocate)
> > is called with deallocate == false
> >
> > But the cache from the test still has zero size after reactivation.
> >
> > Is flag [1] disabled by default because it is not implemented / doesn't
> > work? Do we need to skip it in current ticket and rise new one?
> >
> > ср, 5 февр. 2020 г. в 21:05, Denis Magda :
> >
> >> I believe there might be a consistency-related reason why this flag was
> >> disabled by default for caches that store data in Ignite native
> >> persistence. I hope, Alex Goncharuk or Scherbakov can shed some light on
> >> this.
> >>
> >> As for the memory-only caches or caches backed up by a CacheStore such
> as
> >> an RDBMS, enabling of the flag should be harmless. Once we do this,
> we'll
> >> eliminate the need to load the data back into the cluster which can be a
> >> time-consuming operation depending on the data volume.
> >>
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Wed, Feb 5, 2020 at 11:58 AM Vladimir Steshin
> >> wrote:
> >>
> >> > Denis, but why reuse-data-on-deactivate was disabled by default? There
> >> > should be a reason for that. Any data consistency issues when node
> gets
> >> > activated anew? We may use both solutions because the flag can be
> >> switched
> >> > off again.
> >> >
> >> > ср, 5 февр. 2020 г. в 20:47, Denis Magda :
> >> >
> >> > > Hi Vladimir,
> >> > >
> >> > > Yes, I'm suggesting us to enable this flag by
> >> > >
> >> >
> >>
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> >> > > default instead of introducing --force flag and showing any
> warnings.
> >> > >
> >> > > -
> >> > > Denis
> >> > >
> >> > >
> >> > > On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin >
> >> > > wrote:
> >> > >
> >> > > > Hello all.
> >> > > >
> >> > > > Denis, which changes exactly? In current implementation of ticket
> >> [2]
> >> > > flag
> >> > > > [1] is checked before requiring --force flag and showing any
> >> warnings.
> >> > Do
> >> > > > we need to set reuse-memory-on-deactivate to true by default?
> >> > > >
> >> > > > [1]
> >> > > >
> >> > >
> >> >
> >>
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> >> > > >
> >> > > > [2] https://issues.apache.org/jira/browse/IGNITE-12614
> >> > > >
> >> > > >
> >> > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda :
> >> > > >
> >> > > > > That's the best solution for this scenario. Should we readjust
> the
> >> > > > already
> >> > > > > created ticket [1] suggesting to implement the changes of Alex
> >> > > Scherbakov
> >> > > > > instead?
> >> > > > >
> >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614
> >> > > > >
> >> > > > > -
> >> > > > > Denis
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov <
> >> > > > > alexey.scherbak...@gmail.com> wrote:
> >> > > > >
> >> > > > > > Folks,
> >> > > > > >
> >> > > > > > For a long time we have a flag [1]
> >> >
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Nick
I see no difficulty in assinging each cancellable object an IgniteUuid
(which is actually java UUID and is guaranteed to be unique per node).
We can have registry of running queries and just poking to what registry is
enough to understand query type.
чт, 6 февр. 2020 г. в 17:17, Вячеслав Коптилин :
> Hello,
>
> It seems to me we missed API that should be introduced into control
> utility.
> Nikolay, could you please note this requirement on the IEP page?
>
> Thanks,
> S.
>
> чт, 6 февр. 2020 г. в 15:29, Nikolay Izhikov :
>
> > Ticket [1] created.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12632
> >
> > > 5 февр. 2020 г., в 15:36, Nikolay Izhikov
> > написал(а):
> > >
> > > Alexey.
> > >
> > > I’m talking the following scenario:
> > >
> > > 1. Consider we have unified bean to kill tasks:
> > >
> > > CancelMXBean {
> > > public void cancel(long id);
> > > }
> > >
> > > 2. And we have the following log:
> > >
> > > ```
> > > Transaction with ID=42 started.
> > > Compute task with ID=43 started.
> > > ```
> > >
> > > 3. We want to kill compute task and by mistake executing:
> > >
> > > cancelMxBean.cancel(42); //This will kill transaction not compute task.
> > >
> > > The user doesn’t have a chance to know, what type of object he is
> > killing.
> > > I think we should prevent this type of error by the API design.
> > >
> > >
> > >> 5 февр. 2020 г., в 14:43, Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> > написал(а):
> > >>
> > >> Nikolay,
> > >>
> > >>
> > >>> Consider copy-pasting wrong id from log to its
> > >>> parameters(killing not the buggy compute task, but *VERY* important
> > >>> transaction.
> > >>> How users even know about this type of error with the
> > >>> single method approach?
> > >>>
> > >>> I thought that the identifiers would never intersect (meaning that a
> > >> transaction and a task would never share the same ID)
> > >>
> > >> I agree that change ID types for all objects would be a hard task, so
> > >> probably it's worth discussing a single cancel entry on phase 3.
> > >
> >
> >
>
--
Best regards,
Alexei Scherbakov
Re: Mark MVCC with @IgniteExperimental
I'm strongly support removal of MVCC from master. At the current state it bloats code base and should be reworked from scratch using separate code base. чт, 6 февр. 2020 г. в 19:45, Ilya Kasnacheev : > Hello! > > Please keep in mind that you need to create a separate proposal voting > thread if you really like it to count. I wonder if Dmitry Pavlov can help > us with the procedure. > > Otherwise, I think it makes total sense to restrict MVCC clusters to only > have MVCC caches or REPLICATED TRANSACTIONAL caches (are they compatible in > our current implementation) and no ATOMIC caches at all. > > Regards, > -- > Ilya Kasnacheev > > > чт, 6 февр. 2020 г. в 19:41, Maxim Muzafarov : > > > Ilya, > > > > > > 1. MVCC support requires code maintenance for other developed features > > even if has not used and disabled. Currently, we've got an x2 level of > > difficulty for implementation of new features. > > > > 2. It would be much easy to develop and support clusters with > > mvcc-caches only rather than have a mixed configuration. With this > > option we can dramatically reduce the amount of codebase removing from > > mvcc-branch local, atomic, tx caches. > > > > > > So, I'm +1 to remove it from the master branch and mark the current > > API with @IgniteExperimental. > > > > On Thu, 6 Feb 2020 at 19:29, Ilya Kasnacheev > > wrote: > > > > > > Hello! > > > > > > Why would we drop MVCC!? > > > > > > I can totally imagine a scenario where a large Ignite user surfaces > with > > > fixes for MVCC mode, if it is kept as an experimental feature. Then > maybe > > > it will graduate to beta some time in future. > > > > > > If it does too much strain on the TC, let's discuss that, but I don't > > > remember problems with TC load lately, so maybe this is a moot point. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > чт, 6 февр. 2020 г. в 15:27, Nikolay Izhikov : > > > > > > > > By the way, is there any reason to have this code in the master > > branch > > > > > > > > I support removal MVCC from master. > > > > > > > > > > > > > 6 февр. 2020 г., в 15:26, Andrey Gura > написал(а): > > > > > > > > > > +1 > > > > > > > > > > By the way, is there any reason to have this code in the master > > > > > branch? It seems feature is in unsupportable state and just wastes > TC > > > > > time and our effort to support MVCC related tests. > > > > > > > > > > On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk > > > > > wrote: > > > > >> > > > > >> Agree, let's mark MVCC experimental. > > > > >> > > > > >> Stephen, the annotation serves as an additional > documentation-style > > > > marker. > > > > >> For now there are no compile-time warnings when the API is used. > > > > >> > > > > >> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < > > > > >> stephen.darling...@gridgain.com>: > > > > >> > > > > >>> Yes! I’ve already seen people try to use this without awareness > > that > > > > it’s > > > > >>> not production ready. > > > > >>> > > > > >>> What happens with the annotation, incidentally? Is it just in the > > > > >>> documentation or do you get a compile-time warning? > > > > >>> > > > > On 6 Feb 2020, at 11:32, Nikolay Izhikov > > wrote: > > > > > > > > Hello, Igniters. > > > > > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > > > > > We explicitly note users that MVCC has beta status, for now [1] > > > > > > > > > Beta version of Transactional SQL and MVCC > > > > > In Ignite v2.7, Transactional SQL and MVCC are released as beta > > > > >>> versions to allow users to experiment and share feedback. > > > > > This version of Transactional SQL and MVCC should not be > > considered > > > > for > > > > >>> production. > > > > > > > > > > > > [1] > > > > https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > > > > > > > >>> > > > > >>> > > > > >>> > > > > > > > > > > > -- Best regards, Alexei Scherbakov
Re: [DISCUSS] Links to commercial version resources in community wiki
Hi, Maxim. I had not a chance to start working on it. GG-17396 is about bitset implementation approach for out-of-order updates. Doesn't seem high priority for me right now. чт, 6 февр. 2020 г. в 20:24, Maxim Muzafarov : > Hello, Alexei, > > > Do we have any updates on this issue [1] with replacing GG-X TODOs > by Ignite analogues? > Can you at least share an information about this issue - GG-17396? Is > it still actual? Mentioned here [2]. > > > [1] https://issues.apache.org/jira/browse/IGNITE-12422 > [2] > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/PartitionUpdateCounter.java#L108 > > On Fri, 6 Dec 2019 at 10:55, Alexei Scherbakov > wrote: > > > > https://issues.apache.org/jira/browse/IGNITE-12422 > > > > пт, 6 дек. 2019 г. в 10:27, Alexei Scherbakov < > alexey.scherbak...@gmail.com > > >: > > > > > Maxim, > > > > > > Yes, I think I could help. > > > > > > чт, 5 дек. 2019 г. в 19:52, Maxim Muzafarov : > > > > > >> Alexey, > > >> > > >> Thank you so much! Can you help with the other issues? > > >> It would be great to clean up them and enable checkstyle pattern for > > >> such TODOs [1]. > > >> > > >> Here is the full list of issues: > > >> > > >> GG-1075 > > >> GG-1314 > > >> GG-2311 > > >> GG-2329 > > >> GG-2852 > > >> GG-3946 > > >> GG-3997 > > >> GG-4368 > > >> GG-4440 > > >> GG-4911 > > >> GG-6460 > > >> GG-6879 > > >> GG-8123 > > >> GG-8768 > > >> GG-8884 > > >> GG-9141 > > >> GG-10884 > > >> GG-11141 > > >> GG-11148 > > >> GG-11220 > > >> GG-11241 > > >> GG-11480 > > >> GG-11640 > > >> GG-11768 > > >> GG-12292 > > >> GG-13223 > > >> GG-17396 > > >> > > >> > > >> [1] https://checkstyle.sourceforge.io/config_misc.html#TodoComment > > >> > > >> On Thu, 5 Dec 2019 at 19:18, Alexei Scherbakov > > >> wrote: > > >> > > > >> > This is a contribution artifact and was not done intentionally. > > >> > I'll remove it in subsequent contributions. > > >> > > > >> > ср, 4 дек. 2019 г. в 16:57, Dmitriy Pavlov : > > >> > > > >> > > I agree that 'TODO: please, fix me GG-X' Should be definitely > > >> avoided > > >> > > in the code in all cases. A number of community members don't have > > >> access > > >> > > to these tickets. > > >> > > > > >> > > This is the same as linking to GG's private documentation or > internal > > >> > > network servers. It doesn't make sense. > > >> > > > > >> > > ср, 4 дек. 2019 г. в 14:53, Ivan Pavlukhin : > > >> > > > > >> > > > Maxim, > > >> > > > > > >> > > > References to GG tickets have definitely no meaning for > community. > > >> But > > >> > > > references to documentation on the other hand can be very > useful, > > >> > > > actually it is very similar to references to Wikipedia or > > >> > > > aforementioned Oracle resources. > > >> > > > > > >> > > > ср, 4 дек. 2019 г. в 14:40, Maxim Muzafarov >: > > >> > > > > > > >> > > > > Folks, > > >> > > > > > > >> > > > > In general case, 3rd party resource links look for me > absolutely > > >> > > > > normal e.g. we could have some links on Oracle's documentation > > >> pages. > > >> > > > > > > >> > > > > But let's add a little accuracy. We are talking about links to > > >> > > > > GridGain documentation pages. For me, it's the same as > comments > > >> `TODO: > > >> > > > > please, fix me GG-X` in the source code (e.g. like this > one > > >> [1]). > > >> > > > > Looks a bit strange :-) > > >> > > > > > > >> > > > > > > >> > > > > [1] > > >> > > > > > >> > > > > >> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/PartitionUpdateCounter.java#L43 > > >> > > > > > > >> > > > > On Tue, 3 Dec 2019 at 20:13, Dmitriy Pavlov < > dpav...@apache.org> > > >> > > wrote: > > >> > > > > > > > >> > > > > > Hi Igniters, > > >> > > > > > > > >> > > > > > It is not good if we'll made a tradition of linking to > company > > >> > > websites > > >> > > > > > from the community website / wiki / lists. But if it makes > > >> sense to > > >> > > > mention > > >> > > > > > something useful for users, why not. > > >> > > > > > > > >> > > > > > And in this particular case it may be ok, if it is relevant. > > >> Just > > >> > > > remember, > > >> > > > > > community resources are not Nascar cars. > > >> > > > > > > > >> > > > > > See also https://www.apache.org/foundation/marks/linking > > >> > > > > > > > >> > > > > > Sincerely, > > >> > > > > > Dmitriy Pavlov > > >> > > > > > > > >> > > > > > вт, 3 дек. 2019 г. в 19:50, Denis Magda >: > > >> > > > > > > > >> > > > > > > Hi Ivan, > > >> > > > > > > > > >> > > > > > > If an author of an article believes that there is a 3rd > party > > >> > > > resource that > > >> > > > > > > is handy and helpful for his/her work then there is > nothing > > >> wrong > > >> > > to > > >> > > > refer > > >> > > > > > > to it. Based on Alexey's response in another thread, that > > >> GridGain > > >> > > > page > > >> > > > > > > brings more clarity to the topic, thus, don't
Re: [DISCUSS] Links to commercial version resources in community wiki
Hello, Alexei, Do we have any updates on this issue [1] with replacing GG-X TODOs by Ignite analogues? Can you at least share an information about this issue - GG-17396? Is it still actual? Mentioned here [2]. [1] https://issues.apache.org/jira/browse/IGNITE-12422 [2] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/PartitionUpdateCounter.java#L108 On Fri, 6 Dec 2019 at 10:55, Alexei Scherbakov wrote: > > https://issues.apache.org/jira/browse/IGNITE-12422 > > пт, 6 дек. 2019 г. в 10:27, Alexei Scherbakov >: > > > Maxim, > > > > Yes, I think I could help. > > > > чт, 5 дек. 2019 г. в 19:52, Maxim Muzafarov : > > > >> Alexey, > >> > >> Thank you so much! Can you help with the other issues? > >> It would be great to clean up them and enable checkstyle pattern for > >> such TODOs [1]. > >> > >> Here is the full list of issues: > >> > >> GG-1075 > >> GG-1314 > >> GG-2311 > >> GG-2329 > >> GG-2852 > >> GG-3946 > >> GG-3997 > >> GG-4368 > >> GG-4440 > >> GG-4911 > >> GG-6460 > >> GG-6879 > >> GG-8123 > >> GG-8768 > >> GG-8884 > >> GG-9141 > >> GG-10884 > >> GG-11141 > >> GG-11148 > >> GG-11220 > >> GG-11241 > >> GG-11480 > >> GG-11640 > >> GG-11768 > >> GG-12292 > >> GG-13223 > >> GG-17396 > >> > >> > >> [1] https://checkstyle.sourceforge.io/config_misc.html#TodoComment > >> > >> On Thu, 5 Dec 2019 at 19:18, Alexei Scherbakov > >> wrote: > >> > > >> > This is a contribution artifact and was not done intentionally. > >> > I'll remove it in subsequent contributions. > >> > > >> > ср, 4 дек. 2019 г. в 16:57, Dmitriy Pavlov : > >> > > >> > > I agree that 'TODO: please, fix me GG-X' Should be definitely > >> avoided > >> > > in the code in all cases. A number of community members don't have > >> access > >> > > to these tickets. > >> > > > >> > > This is the same as linking to GG's private documentation or internal > >> > > network servers. It doesn't make sense. > >> > > > >> > > ср, 4 дек. 2019 г. в 14:53, Ivan Pavlukhin : > >> > > > >> > > > Maxim, > >> > > > > >> > > > References to GG tickets have definitely no meaning for community. > >> But > >> > > > references to documentation on the other hand can be very useful, > >> > > > actually it is very similar to references to Wikipedia or > >> > > > aforementioned Oracle resources. > >> > > > > >> > > > ср, 4 дек. 2019 г. в 14:40, Maxim Muzafarov : > >> > > > > > >> > > > > Folks, > >> > > > > > >> > > > > In general case, 3rd party resource links look for me absolutely > >> > > > > normal e.g. we could have some links on Oracle's documentation > >> pages. > >> > > > > > >> > > > > But let's add a little accuracy. We are talking about links to > >> > > > > GridGain documentation pages. For me, it's the same as comments > >> `TODO: > >> > > > > please, fix me GG-X` in the source code (e.g. like this one > >> [1]). > >> > > > > Looks a bit strange :-) > >> > > > > > >> > > > > > >> > > > > [1] > >> > > > > >> > > > >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/PartitionUpdateCounter.java#L43 > >> > > > > > >> > > > > On Tue, 3 Dec 2019 at 20:13, Dmitriy Pavlov > >> > > wrote: > >> > > > > > > >> > > > > > Hi Igniters, > >> > > > > > > >> > > > > > It is not good if we'll made a tradition of linking to company > >> > > websites > >> > > > > > from the community website / wiki / lists. But if it makes > >> sense to > >> > > > mention > >> > > > > > something useful for users, why not. > >> > > > > > > >> > > > > > And in this particular case it may be ok, if it is relevant. > >> Just > >> > > > remember, > >> > > > > > community resources are not Nascar cars. > >> > > > > > > >> > > > > > See also https://www.apache.org/foundation/marks/linking > >> > > > > > > >> > > > > > Sincerely, > >> > > > > > Dmitriy Pavlov > >> > > > > > > >> > > > > > вт, 3 дек. 2019 г. в 19:50, Denis Magda : > >> > > > > > > >> > > > > > > Hi Ivan, > >> > > > > > > > >> > > > > > > If an author of an article believes that there is a 3rd party > >> > > > resource that > >> > > > > > > is handy and helpful for his/her work then there is nothing > >> wrong > >> > > to > >> > > > refer > >> > > > > > > to it. Based on Alexey's response in another thread, that > >> GridGain > >> > > > page > >> > > > > > > brings more clarity to the topic, thus, don't see any issue > >> with > >> > > > that. > >> > > > > > > > >> > > > > > > > >> > > > > > > - > >> > > > > > > Denis > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tue, Dec 3, 2019 at 6:41 AM Ivan Pavlukhin < > >> vololo...@gmail.com > >> > > > > >> > > > wrote: > >> > > > > > > > >> > > > > > > > Folks, > >> > > > > > > > > >> > > > > > > > In a thread about a document describing data consistency > >> [1] in > >> > > > Ignite > >> > > > > > > > it was identified that there is a link to GridGain > >> documentation. > >> > > > > > > > > >> > > > > > > > I do not have strong arguments
Re: [DISCUSS] Public API deprecation rules
Ivan, thanks, I agree, added this clause: The vote we are going to have is reduced to two choices so far: * Never deprecate the old APIs unless the new APIs are stable and released without @IgniteExperimental. The old APIs javadoc may be updated with a reference to new APIs to encourage users to evaluate new APIs. The deprecation and new API release may happen simultaneously if the new API is not marked with @IgniteExperimental or the annotation is removed in the same release. * Allow to deprecate the old APIs even when new APIs are marked with @IgniteExperimental to explicitly notify users that old APIs will be removed in the next major release AND new APIs are available Neither of cases prohibits deprecation of an API without a replacement if community decided so. Are we good to go?
Re: Mark MVCC with @IgniteExperimental
Hello! Please keep in mind that you need to create a separate proposal voting thread if you really like it to count. I wonder if Dmitry Pavlov can help us with the procedure. Otherwise, I think it makes total sense to restrict MVCC clusters to only have MVCC caches or REPLICATED TRANSACTIONAL caches (are they compatible in our current implementation) and no ATOMIC caches at all. Regards, -- Ilya Kasnacheev чт, 6 февр. 2020 г. в 19:41, Maxim Muzafarov : > Ilya, > > > 1. MVCC support requires code maintenance for other developed features > even if has not used and disabled. Currently, we've got an x2 level of > difficulty for implementation of new features. > > 2. It would be much easy to develop and support clusters with > mvcc-caches only rather than have a mixed configuration. With this > option we can dramatically reduce the amount of codebase removing from > mvcc-branch local, atomic, tx caches. > > > So, I'm +1 to remove it from the master branch and mark the current > API with @IgniteExperimental. > > On Thu, 6 Feb 2020 at 19:29, Ilya Kasnacheev > wrote: > > > > Hello! > > > > Why would we drop MVCC!? > > > > I can totally imagine a scenario where a large Ignite user surfaces with > > fixes for MVCC mode, if it is kept as an experimental feature. Then maybe > > it will graduate to beta some time in future. > > > > If it does too much strain on the TC, let's discuss that, but I don't > > remember problems with TC load lately, so maybe this is a moot point. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > чт, 6 февр. 2020 г. в 15:27, Nikolay Izhikov : > > > > > > By the way, is there any reason to have this code in the master > branch > > > > > > I support removal MVCC from master. > > > > > > > > > > 6 февр. 2020 г., в 15:26, Andrey Gura написал(а): > > > > > > > > +1 > > > > > > > > By the way, is there any reason to have this code in the master > > > > branch? It seems feature is in unsupportable state and just wastes TC > > > > time and our effort to support MVCC related tests. > > > > > > > > On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk > > > > wrote: > > > >> > > > >> Agree, let's mark MVCC experimental. > > > >> > > > >> Stephen, the annotation serves as an additional documentation-style > > > marker. > > > >> For now there are no compile-time warnings when the API is used. > > > >> > > > >> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < > > > >> stephen.darling...@gridgain.com>: > > > >> > > > >>> Yes! I’ve already seen people try to use this without awareness > that > > > it’s > > > >>> not production ready. > > > >>> > > > >>> What happens with the annotation, incidentally? Is it just in the > > > >>> documentation or do you get a compile-time warning? > > > >>> > > > On 6 Feb 2020, at 11:32, Nikolay Izhikov > wrote: > > > > > > Hello, Igniters. > > > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > > > We explicitly note users that MVCC has beta status, for now [1] > > > > > > > Beta version of Transactional SQL and MVCC > > > > In Ignite v2.7, Transactional SQL and MVCC are released as beta > > > >>> versions to allow users to experiment and share feedback. > > > > This version of Transactional SQL and MVCC should not be > considered > > > for > > > >>> production. > > > > > > > > > [1] > > > https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > > > > >>> > > > >>> > > > >>> > > > > > > >
[jira] [Created] (IGNITE-12637) IgniteSparkSession doesn't start the clients on really distributed cluster
Andrey Aleksandrov created IGNITE-12637:
---
Summary: IgniteSparkSession doesn't start the clients on really
distributed cluster
Key: IGNITE-12637
URL: https://issues.apache.org/jira/browse/IGNITE-12637
Project: Ignite
Issue Type: Bug
Components: spark
Affects Versions: 2.7.6
Reporter: Andrey Aleksandrov
Fix For: 2.8
Next code:
IgniteSparkSession igniteSession = IgniteSparkSession.builder()
.appName("Spark Ignite example")
.igniteConfig(configPath)
.getOrCreate();
Throws:
class org.apache.ignite.IgniteIllegalStateException: Ignite instance with
provided name doesn't exist. Did you call Ignition.start(..) to start an Ignite
instance? [name=grid]
Client config was located in all Spark nodes at the same place.
When I ran these tests on the same host with several local executors then it
worked. But if executors were located on different hosts then it didn't.
DataFrame API works fine with the same config.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Mark MVCC with @IgniteExperimental
Ilya, 1. MVCC support requires code maintenance for other developed features even if has not used and disabled. Currently, we've got an x2 level of difficulty for implementation of new features. 2. It would be much easy to develop and support clusters with mvcc-caches only rather than have a mixed configuration. With this option we can dramatically reduce the amount of codebase removing from mvcc-branch local, atomic, tx caches. So, I'm +1 to remove it from the master branch and mark the current API with @IgniteExperimental. On Thu, 6 Feb 2020 at 19:29, Ilya Kasnacheev wrote: > > Hello! > > Why would we drop MVCC!? > > I can totally imagine a scenario where a large Ignite user surfaces with > fixes for MVCC mode, if it is kept as an experimental feature. Then maybe > it will graduate to beta some time in future. > > If it does too much strain on the TC, let's discuss that, but I don't > remember problems with TC load lately, so maybe this is a moot point. > > Regards, > -- > Ilya Kasnacheev > > > чт, 6 февр. 2020 г. в 15:27, Nikolay Izhikov : > > > > By the way, is there any reason to have this code in the master branch > > > > I support removal MVCC from master. > > > > > > > 6 февр. 2020 г., в 15:26, Andrey Gura написал(а): > > > > > > +1 > > > > > > By the way, is there any reason to have this code in the master > > > branch? It seems feature is in unsupportable state and just wastes TC > > > time and our effort to support MVCC related tests. > > > > > > On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk > > > wrote: > > >> > > >> Agree, let's mark MVCC experimental. > > >> > > >> Stephen, the annotation serves as an additional documentation-style > > marker. > > >> For now there are no compile-time warnings when the API is used. > > >> > > >> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < > > >> stephen.darling...@gridgain.com>: > > >> > > >>> Yes! I’ve already seen people try to use this without awareness that > > it’s > > >>> not production ready. > > >>> > > >>> What happens with the annotation, incidentally? Is it just in the > > >>> documentation or do you get a compile-time warning? > > >>> > > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > > > Hello, Igniters. > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > We explicitly note users that MVCC has beta status, for now [1] > > > > > Beta version of Transactional SQL and MVCC > > > In Ignite v2.7, Transactional SQL and MVCC are released as beta > > >>> versions to allow users to experiment and share feedback. > > > This version of Transactional SQL and MVCC should not be considered > > for > > >>> production. > > > > > > [1] > > https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > >>> > > >>> > > >>> > > > >
Re: Mark MVCC with @IgniteExperimental
Hello! Why would we drop MVCC!? I can totally imagine a scenario where a large Ignite user surfaces with fixes for MVCC mode, if it is kept as an experimental feature. Then maybe it will graduate to beta some time in future. If it does too much strain on the TC, let's discuss that, but I don't remember problems with TC load lately, so maybe this is a moot point. Regards, -- Ilya Kasnacheev чт, 6 февр. 2020 г. в 15:27, Nikolay Izhikov : > > By the way, is there any reason to have this code in the master branch > > I support removal MVCC from master. > > > > 6 февр. 2020 г., в 15:26, Andrey Gura написал(а): > > > > +1 > > > > By the way, is there any reason to have this code in the master > > branch? It seems feature is in unsupportable state and just wastes TC > > time and our effort to support MVCC related tests. > > > > On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk > > wrote: > >> > >> Agree, let's mark MVCC experimental. > >> > >> Stephen, the annotation serves as an additional documentation-style > marker. > >> For now there are no compile-time warnings when the API is used. > >> > >> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < > >> stephen.darling...@gridgain.com>: > >> > >>> Yes! I’ve already seen people try to use this without awareness that > it’s > >>> not production ready. > >>> > >>> What happens with the annotation, incidentally? Is it just in the > >>> documentation or do you get a compile-time warning? > >>> > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > Hello, Igniters. > > Should we mark MVCC feature with the new @IgniteExperimental? > > We explicitly note users that MVCC has beta status, for now [1] > > > Beta version of Transactional SQL and MVCC > > In Ignite v2.7, Transactional SQL and MVCC are released as beta > >>> versions to allow users to experiment and share feedback. > > This version of Transactional SQL and MVCC should not be considered > for > >>> production. > > > [1] > https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > >>> > >>> > >>> > >
Re: Question: different setting for nodes.
Hello! How about we introduce a *developer warning* about this inconsistency, see what happens in a release with this warning, and then maybe eventually turn it into error? Please note that plain error will be breaking existing code which should make it blocker for minor releases. Regards, -- Ilya Kasnacheev ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov : > Folks, > > > I think it's a common problem for such system properties (which may be > different on different nodes and which may lead to unpredictable > cluster behaviour). The same mentioned by Ivan here [1]. > > Is it better to validate (somehow) all system properties, for > instance, started with IGNITE_ prefix, on all nodes the same way as we > do it for IgniteConfiguration? > > [1] > https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207 > > On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov wrote: > > > > Vladimir. > > > > This looks like a bug to me. > > > > Can you, please, prepare the simple reproducer for this issue and it’s > consequences? > > > > > > > 5 февр. 2020 г., в 17:08, Vladimir Steshin > написал(а): > > > > > > Hi, folks. > > > > > > > > > > > > I recently found that one node might be started with flag > > > reuse-memory-on-deactivate [1] set off while another node might be > > > configured with the flag enabled. This ability hinders prediction of > > > cluster behavior on deactivation/activation. One node keeps data in > memory, > > > other doesn’t. > > > > > > Should we check this flag is set equally on all nodes? Is this an > issue? We > > > might prevent node start if it has different setting. > > > > > > > > > > > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > >
[jira] [Created] (IGNITE-12636) Full rebalance instead of historical one
Mirza Aliev created IGNITE-12636:
Summary: Full rebalance instead of historical one
Key: IGNITE-12636
URL: https://issues.apache.org/jira/browse/IGNITE-12636
Project: Ignite
Issue Type: Bug
Reporter: Mirza Aliev
Assignee: Mirza Aliev
Common configuration:
1)4 nodes
2)3-4 caches
3)IGNITE_PDS_WAL_REBALANCE_THRESHOLD=500
4)walHistorySize=500
5)IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE=500
Scenario:
1)Load a lot of data.
2)Start the transaction on some client but DON'T close it.
3)Stop the server node.
4)Start the server.
5)PME should be started
6)Kill the client started the transaction
Expected result:
Rebalance is HISTORICAL.
Actual result:
Rebalance is full:
{{[12:01:58,613][INFO][sys-#95][GridDhtPartitionDemander] Started rebalance
routine [cache_group_6, supplier=5462dc46-f71f-49d8-8a1d-d9d69c3e372a, topic=0,
fullPartitions=[23], histPartitions=[]]
[12:01:58,614][INFO][sys-#109][GridDhtPartitionDemander] Started rebalance
routine [cache_group_6, supplier=8ab78982-0bcf-494f-a634-f3fb2d78328f, topic=0,
fullPartitions=[1], histPartitions=[]]
[12:01:58,614][INFO][sys-#101][GridDhtPartitionDemander] Started rebalance
routine [cache_group_6, supplier=8ab78982-0bcf-494f-a634-f3fb2d78328f, topic=1,
fullPartitions=[55], histPartitions=[]]
[12:01:59,004][INFO][sys-#99][GridDhtPartitionDemander] Started rebalance
routine [cache_group_4_118, supplier=5462dc46-f71f-49d8-8a1d-d9d69c3e372a,
topic=0, fullPartitions=[5], histPartitions=[]]
[12:01:59,004][INFO][sys-#96][GridDhtPartitionDemander] Started rebalance
routine [cache_group_4_118, supplier=48e2a2b5-2119-4b5c-873c-eb8d0c436b6a,
topic=0, fullPartitions=[15], histPartitions=[]]
[12:01:59,196][INFO][sys-#104][GridDhtPartitionDemander] Started rebalance
routine [cache_group_2_058, supplier=48e2a2b5-2119-4b5c-873c-eb8d0c436b6a,
topic=0, fullPartitions=[21], histPartitions=[]]
[12:01:59,196][INFO][sys-#95][GridDhtPartitionDemander] Started rebalance
routine [cache_group_2_058, supplier=8ab78982-0bcf-494f-a634-f3fb2d78328f,
topic=0, fullPartitions=[19], histPartitions=[]]}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Public API deprecation rules
A little bit more formalism. The thread subject is about API deprecation in general. But it looks like that discussion boils down to a replacement of old API with a new one. I suppose there should be a possibility to deprecate API for removal if the Community decided to do so. > @IgniteExperimental [3] which marks APIs which are yet not stable, dangerous, > partially implemented or are allowed to be changed in a future. "Dangerous" part is not clearly stated in javadocs [1]. But I tend to think about "experimental" features as not-production-ready so possibly with dangerous bugs. I suppose that there can be another flavor of API annotations like "UntableAPI" (or non finalized or some better term) which can be used when an underlying feature is production-ready but an API can be changed without preserving backwards compatibility. We might continue discussion in the thread about experimental API annotations. [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/lang/IgniteExperimental.java Best regards, Ivan Pavlukhin чт, 6 февр. 2020 г. в 14:50, Alexey Goncharuk : > > Thanks, updated the justification: > > The vote we are going to have is reduced to two choices so far: > * Never deprecate the old APIs unless the new APIs are stable and released > without @IgniteExperimental. The old APIs javadoc may be updated with a > reference to new APIs to encourage users to evaluate new APIs. The > deprecation and new API release may happen simultaneously if the new API is > not marked with @IgniteExperimental or the annotation is removed in the > same release. > * Allow to deprecate the old APIs even when new APIs are marked with > @IgniteExperimental to explicitly notify users that old APIs will be > removed in the next major release AND new APIs are available > > As for the @IgniteExperimental precise meaning - I do not agree that the > experimental essence is related to the API only. As you suggested in the > separate email, we are going to mark MVCC as experimental because it is in > beta meaning that there are currently known scenarios when the new feature > breaks or simply is not implemented. Yet the API for MVCC cache is quite > stable and adds nothing more than a separate cache mode. Not sure if this > changes the vote, though.
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Hello,
It seems to me we missed API that should be introduced into control utility.
Nikolay, could you please note this requirement on the IEP page?
Thanks,
S.
чт, 6 февр. 2020 г. в 15:29, Nikolay Izhikov :
> Ticket [1] created.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12632
>
> > 5 февр. 2020 г., в 15:36, Nikolay Izhikov
> написал(а):
> >
> > Alexey.
> >
> > I’m talking the following scenario:
> >
> > 1. Consider we have unified bean to kill tasks:
> >
> > CancelMXBean {
> > public void cancel(long id);
> > }
> >
> > 2. And we have the following log:
> >
> > ```
> > Transaction with ID=42 started.
> > Compute task with ID=43 started.
> > ```
> >
> > 3. We want to kill compute task and by mistake executing:
> >
> > cancelMxBean.cancel(42); //This will kill transaction not compute task.
> >
> > The user doesn’t have a chance to know, what type of object he is
> killing.
> > I think we should prevent this type of error by the API design.
> >
> >
> >> 5 февр. 2020 г., в 14:43, Alexey Goncharuk
> написал(а):
> >>
> >> Nikolay,
> >>
> >>
> >>> Consider copy-pasting wrong id from log to its
> >>> parameters(killing not the buggy compute task, but *VERY* important
> >>> transaction.
> >>> How users even know about this type of error with the
> >>> single method approach?
> >>>
> >>> I thought that the identifiers would never intersect (meaning that a
> >> transaction and a task would never share the same ID)
> >>
> >> I agree that change ID types for all objects would be a hard task, so
> >> probably it's worth discussing a single cancel entry on phase 3.
> >
>
>
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Or, is flag [1] is actual only for persistence mode? Related ticket [2] is
completely about disabled persistence.
If previous assumption is right, I think, we can involve flag [1] in ticket
[2].
[1]
org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
[2] https://issues.apache.org/jira/browse/IGNITE-12614
чт, 6 февр. 2020 г. в 16:10, Vladimir Steshin :
> Denis, Alexei,
>
> Regarding usage of flag
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> [1]
>
> When enabled, I think the following test should work. But it fails.
>
> //
> @Test
> public void testDataPresent() throws Exception {
> System.setProperty(IGNITE_REUSE_MEMORY_ON_DEACTIVATE, "true");
>
> IgniteEx i = startGrid(0);
>
> assertFalse(
> i.configuration().getDataStorageConfiguration().getDefaultDataRegionConfiguration()
> .isPersistenceEnabled() );
>
> String name = "non-persistent-cache";
>
> i.createCache(name).put(1L, 1L);
>
> assertEquals(1L, i.cache(name).get(1L));
>
> i.cluster().state(ClusterState.INACTIVE);
> i.cluster().state(ClusterState.ACTIVE);
>
> assertEquals(1L, i.cache(name).get(1L)); //Assertion error here!
> }
> //
>
>
> Several notes:
>
> - IgniteCacheDatabaseSharedManager#reuseMemory
> is true
> - IgniteCacheDatabaseSharedManager#onDeActivate(boolean shutdown)
> is called with shutdown == false
> - PageMemoryNoStoreImpl#stop(booleam deallocate)
> is called with deallocate == false
>
> But the cache from the test still has zero size after reactivation.
>
> Is flag [1] disabled by default because it is not implemented / doesn't
> work? Do we need to skip it in current ticket and rise new one?
>
> ср, 5 февр. 2020 г. в 21:05, Denis Magda :
>
>> I believe there might be a consistency-related reason why this flag was
>> disabled by default for caches that store data in Ignite native
>> persistence. I hope, Alex Goncharuk or Scherbakov can shed some light on
>> this.
>>
>> As for the memory-only caches or caches backed up by a CacheStore such as
>> an RDBMS, enabling of the flag should be harmless. Once we do this, we'll
>> eliminate the need to load the data back into the cluster which can be a
>> time-consuming operation depending on the data volume.
>>
>>
>> -
>> Denis
>>
>>
>> On Wed, Feb 5, 2020 at 11:58 AM Vladimir Steshin
>> wrote:
>>
>> > Denis, but why reuse-data-on-deactivate was disabled by default? There
>> > should be a reason for that. Any data consistency issues when node gets
>> > activated anew? We may use both solutions because the flag can be
>> switched
>> > off again.
>> >
>> > ср, 5 февр. 2020 г. в 20:47, Denis Magda :
>> >
>> > > Hi Vladimir,
>> > >
>> > > Yes, I'm suggesting us to enable this flag by
>> > >
>> >
>> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
>> > > default instead of introducing --force flag and showing any warnings.
>> > >
>> > > -
>> > > Denis
>> > >
>> > >
>> > > On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin
>> > > wrote:
>> > >
>> > > > Hello all.
>> > > >
>> > > > Denis, which changes exactly? In current implementation of ticket
>> [2]
>> > > flag
>> > > > [1] is checked before requiring --force flag and showing any
>> warnings.
>> > Do
>> > > > we need to set reuse-memory-on-deactivate to true by default?
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
>> > > >
>> > > > [2] https://issues.apache.org/jira/browse/IGNITE-12614
>> > > >
>> > > >
>> > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda :
>> > > >
>> > > > > That's the best solution for this scenario. Should we readjust the
>> > > > already
>> > > > > created ticket [1] suggesting to implement the changes of Alex
>> > > Scherbakov
>> > > > > instead?
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614
>> > > > >
>> > > > > -
>> > > > > Denis
>> > > > >
>> > > > >
>> > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov <
>> > > > > alexey.scherbak...@gmail.com> wrote:
>> > > > >
>> > > > > > Folks,
>> > > > > >
>> > > > > > For a long time we have a flag [1]
>> > > > > >
>> > > > > > It does almost what we want here.
>> > > > > >
>> > > > > > I suggest to make this behavior default and rework it to keep
>> data
>> > in
>> > > > > > memory as well (we already have special "recovery" mode for
>> > caches).
>> > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk <
>> > > > > alexey.goncha...@gmail.com
>> > > > > > >:
>> > > > > >
>> > > > > > > I do not mind making this change if we explicitly
[jira] [Created] (IGNITE-12635) Reservation WALRecord for compatibility purposes
Yury Gerzhedovich created IGNITE-12635: -- Summary: Reservation WALRecord for compatibility purposes Key: IGNITE-12635 URL: https://issues.apache.org/jira/browse/IGNITE-12635 Project: Ignite Issue Type: Task Components: persistence Reporter: Yury Gerzhedovich Assignee: Yury Gerzhedovich Fix For: 2.8 We need reserve record with id 59 (org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType) for compatibility purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Denis, Alexei,
Regarding usage of flag
org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
[1]
When enabled, I think the following test should work. But it fails.
//
@Test
public void testDataPresent() throws Exception {
System.setProperty(IGNITE_REUSE_MEMORY_ON_DEACTIVATE, "true");
IgniteEx i = startGrid(0);
assertFalse(
i.configuration().getDataStorageConfiguration().getDefaultDataRegionConfiguration()
.isPersistenceEnabled() );
String name = "non-persistent-cache";
i.createCache(name).put(1L, 1L);
assertEquals(1L, i.cache(name).get(1L));
i.cluster().state(ClusterState.INACTIVE);
i.cluster().state(ClusterState.ACTIVE);
assertEquals(1L, i.cache(name).get(1L)); //Assertion error here!
}
//
Several notes:
- IgniteCacheDatabaseSharedManager#reuseMemory
is true
- IgniteCacheDatabaseSharedManager#onDeActivate(boolean shutdown)
is called with shutdown == false
- PageMemoryNoStoreImpl#stop(booleam deallocate)
is called with deallocate == false
But the cache from the test still has zero size after reactivation.
Is flag [1] disabled by default because it is not implemented / doesn't
work? Do we need to skip it in current ticket and rise new one?
ср, 5 февр. 2020 г. в 21:05, Denis Magda :
> I believe there might be a consistency-related reason why this flag was
> disabled by default for caches that store data in Ignite native
> persistence. I hope, Alex Goncharuk or Scherbakov can shed some light on
> this.
>
> As for the memory-only caches or caches backed up by a CacheStore such as
> an RDBMS, enabling of the flag should be harmless. Once we do this, we'll
> eliminate the need to load the data back into the cluster which can be a
> time-consuming operation depending on the data volume.
>
>
> -
> Denis
>
>
> On Wed, Feb 5, 2020 at 11:58 AM Vladimir Steshin
> wrote:
>
> > Denis, but why reuse-data-on-deactivate was disabled by default? There
> > should be a reason for that. Any data consistency issues when node gets
> > activated anew? We may use both solutions because the flag can be
> switched
> > off again.
> >
> > ср, 5 февр. 2020 г. в 20:47, Denis Magda :
> >
> > > Hi Vladimir,
> > >
> > > Yes, I'm suggesting us to enable this flag by
> > >
> >
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> > > default instead of introducing --force flag and showing any warnings.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin
> > > wrote:
> > >
> > > > Hello all.
> > > >
> > > > Denis, which changes exactly? In current implementation of ticket [2]
> > > flag
> > > > [1] is checked before requiring --force flag and showing any
> warnings.
> > Do
> > > > we need to set reuse-memory-on-deactivate to true by default?
> > > >
> > > > [1]
> > > >
> > >
> >
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> > > >
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-12614
> > > >
> > > >
> > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda :
> > > >
> > > > > That's the best solution for this scenario. Should we readjust the
> > > > already
> > > > > created ticket [1] suggesting to implement the changes of Alex
> > > Scherbakov
> > > > > instead?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov <
> > > > > alexey.scherbak...@gmail.com> wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > For a long time we have a flag [1]
> > > > > >
> > > > > > It does almost what we want here.
> > > > > >
> > > > > > I suggest to make this behavior default and rework it to keep
> data
> > in
> > > > > > memory as well (we already have special "recovery" mode for
> > caches).
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
> > > > > >
> > > > > >
> > > > > >
> > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > I do not mind making this change if we explicitly and clearly
> > > define
> > > > > what
> > > > > > > 'new inactive state' means. What should happen if a partition
> is
> > > lost
> > > > > in
> > > > > > > inactive state? What if such node joins back the cluster after?
> > > Etc.
> > > > > > >
> > > > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda :
> > > > > > >
> > > > > > > > Back up Ivan's opinion here. Initially, the
> > > activation/deactivation
> > > > > was
> > > > > > > > created for the baseline topology designed for cases with
> > native
> > > > > > > > persistence. My thinking was that the
[jira] [Created] (IGNITE-12634) .NET: Publish symbol packages
Pavel Tupitsyn created IGNITE-12634: --- Summary: .NET: Publish symbol packages Key: IGNITE-12634 URL: https://issues.apache.org/jira/browse/IGNITE-12634 Project: Ignite Issue Type: Bug Components: platforms Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 3.0 Symbol packages (snupkg) improve debugging experience for the users: https://docs.microsoft.com/en-us/nuget/create-packages/symbol-packages-snupkg Create and publish Ignite.NET symbol packages along with normal NuGet packages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Ticket [1] created.
[1] https://issues.apache.org/jira/browse/IGNITE-12632
> 5 февр. 2020 г., в 15:36, Nikolay Izhikov написал(а):
>
> Alexey.
>
> I’m talking the following scenario:
>
> 1. Consider we have unified bean to kill tasks:
>
> CancelMXBean {
> public void cancel(long id);
> }
>
> 2. And we have the following log:
>
> ```
> Transaction with ID=42 started.
> Compute task with ID=43 started.
> ```
>
> 3. We want to kill compute task and by mistake executing:
>
> cancelMxBean.cancel(42); //This will kill transaction not compute task.
>
> The user doesn’t have a chance to know, what type of object he is killing.
> I think we should prevent this type of error by the API design.
>
>
>> 5 февр. 2020 г., в 14:43, Alexey Goncharuk
>> написал(а):
>>
>> Nikolay,
>>
>>
>>> Consider copy-pasting wrong id from log to its
>>> parameters(killing not the buggy compute task, but *VERY* important
>>> transaction.
>>> How users even know about this type of error with the
>>> single method approach?
>>>
>>> I thought that the identifiers would never intersect (meaning that a
>> transaction and a task would never share the same ID)
>>
>> I agree that change ID types for all objects would be a hard task, so
>> probably it's worth discussing a single cancel entry on phase 3.
>
Re: Mark MVCC with @IgniteExperimental
> By the way, is there any reason to have this code in the master branch I support removal MVCC from master. > 6 февр. 2020 г., в 15:26, Andrey Gura написал(а): > > +1 > > By the way, is there any reason to have this code in the master > branch? It seems feature is in unsupportable state and just wastes TC > time and our effort to support MVCC related tests. > > On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk > wrote: >> >> Agree, let's mark MVCC experimental. >> >> Stephen, the annotation serves as an additional documentation-style marker. >> For now there are no compile-time warnings when the API is used. >> >> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < >> stephen.darling...@gridgain.com>: >> >>> Yes! I’ve already seen people try to use this without awareness that it’s >>> not production ready. >>> >>> What happens with the annotation, incidentally? Is it just in the >>> documentation or do you get a compile-time warning? >>> On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: Hello, Igniters. Should we mark MVCC feature with the new @IgniteExperimental? We explicitly note users that MVCC has beta status, for now [1] > Beta version of Transactional SQL and MVCC > In Ignite v2.7, Transactional SQL and MVCC are released as beta >>> versions to allow users to experiment and share feedback. > This version of Transactional SQL and MVCC should not be considered for >>> production. [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control >>> >>> >>>
Re: Mark MVCC with @IgniteExperimental
+1 By the way, is there any reason to have this code in the master branch? It seems feature is in unsupportable state and just wastes TC time and our effort to support MVCC related tests. On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk wrote: > > Agree, let's mark MVCC experimental. > > Stephen, the annotation serves as an additional documentation-style marker. > For now there are no compile-time warnings when the API is used. > > чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < > stephen.darling...@gridgain.com>: > > > Yes! I’ve already seen people try to use this without awareness that it’s > > not production ready. > > > > What happens with the annotation, incidentally? Is it just in the > > documentation or do you get a compile-time warning? > > > > > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > > > > > Hello, Igniters. > > > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > > > We explicitly note users that MVCC has beta status, for now [1] > > > > > >> Beta version of Transactional SQL and MVCC > > >> In Ignite v2.7, Transactional SQL and MVCC are released as beta > > versions to allow users to experiment and share feedback. > > >> This version of Transactional SQL and MVCC should not be considered for > > production. > > > > > > > > > [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > > > > > > >
Re: Explicit plugin providers configuration issue
> ... > So I think that we can't mix these two approaches. Sounds reasonable. Thanks for clarification. > Also, it's not clear for me -- are there any reasons to continue support > of the ServiceLoader approach, given the fact that configurations needed > for a plugin, in this case, are also specified via IgniteConfiguration? I have two answers ) - No, there are no such reasons in the future (starting from next major release). - Yes, there is at least one reason at present - backward compatibility. > Moreover, why someone should mix those approaches? Actually mixed approach has the right to exist. But isn't matter. Anyway, ServiceLoader based approach requires additional plugin configuration and this is reason for some simplification. On Wed, Feb 5, 2020 at 8:46 PM Mikhail Petrov wrote: > > Hi, Andrey. > > The approach of setting plugins via IgniteConfiguration provides an > additional feature to control the order in which plugins will be loaded. > It's important in the case when several plugins provide different > implementations of the same GridComponent while providing some other > unique implementations (now the first found plugin that provides > required component will be used for its obtaining). > > The ServiceLoader approach gives no mentioned above order guarantees. > > So I think that we can't mix these two approaches. > > Also, it's not clear for me -- are there any reasons to continue support > of the ServiceLoader approach, given the fact that configurations needed > for a plugin, in this case, are also specified via IgniteConfiguration? > Moreover, why someone should mix those approaches? > > On 05.02.2020 17:35, Andrey Gura wrote: > > Anton, Mikhail, > > > > I stumbled upon impossibility to configure plugin with out adding > > appropriate class names to org.apache.ignite.plugin.PluginProvider > > file and found change where this possibility was added [1]. > > > > Thanks a lot for this change. I noticed also that > > IgniteConfiguration.setPluginProviders() overrides plugins that > > usually will be loaded by ServiceLoader. Is it deliberate decision? It > > seems that better approach is using plugins from both sources: > > ServiceLoader provided and user provided via IgniteConfiguration. > > > > WDYT? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-11744
[jira] [Created] (IGNITE-12633) Calcite integration. Query cancel purpose.
Igor Seliverstov created IGNITE-12633: - Summary: Calcite integration. Query cancel purpose. Key: IGNITE-12633 URL: https://issues.apache.org/jira/browse/IGNITE-12633 Project: Ignite Issue Type: Improvement Components: sql Reporter: Igor Seliverstov Currently on a query participant node left whole query is cancelled, but the end user doesn't know the purpose. We should pass causing cancel exception to end user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Permissions
Hi Artsiom, Welcome to Apache Ignite Community! I added a contributor role to your Jira account. Now you can assign tickets to yourself. Do not hesitate to ask if you need any assistance. Please check this page out for commonly asked questions pertaining to the contribution process: https://ignite.apache.org/community/contribute.html Best regards, Ivan Pavlukhin чт, 6 февр. 2020 г. в 13:36, Artsiom Panko : > > Hello! > I want to be a new community member so I ask you to give me contributors > permissions > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=artsiom_panko
[jira] [Created] (IGNITE-12632) [IEP-39] Management API to cancel user provided tasks and queries.
Nikolay Izhikov created IGNITE-12632: Summary: [IEP-39] Management API to cancel user provided tasks and queries. Key: IGNITE-12632 URL: https://issues.apache.org/jira/browse/IGNITE-12632 Project: Ignite Issue Type: Improvement Reporter: Nikolay Izhikov Assignee: Nikolay Izhikov Ignite provides many API to deploy and execute user-provided code on the server nodes inside the same JVM as the Ignite process runs. Ignite has many APIs that allocate many resources on the server nodes, also. In case of some buggy code that consumes many system resources(CPU, RAM, flood network) or heavy query the whole cluster can become unstable. We should provide to the cluster administrator the ability to stop any user deployed task. JMX beans to cancel listed tasks should be introduced: * Compute task * Service * Continuous query * Transactions * Queries(SQL, Scan, Text) In the scope of IEP-35 System view API introduced. A new API should use the same identifier that is used in corresponding System View. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION] ignite-extension - ignite-spring-boot-autoconfigurer release.
Hello, Ivan. > As usual we need to decide about versioning and a correspondence to Ignite > versions > As we are going to have a separate release cycle I can imagine an independent > versioning scheme with a range of supported Ignite versions. Please, clarify, what do you mean? Do you have any idea how it should be implemented? I thought we just release module with the 1.0.0 version and specify supported Ignite version somewhere in the documentation. > 6 февр. 2020 г., в 11:15, Ivan Pavlukhin написал(а): > > Nikolay, > > Thank you for driving it! It is great to establish this process in > practice earlier because it seems that we need to release a Flink > integration soon because Ignite 2.8 is going to be released without > that integration bundled. > > As usual we need to decide about versioning and a correspondence to > Ignite versions. As we are going to have a separate release cycle I > can imagine an independent versioning scheme with a range of supported > Ignite versions. > > Also I think it is a good idea to update ignite-extensions README on > GitHub and provide a link to relevant TC jobs. > > Best regards, > Ivan Pavlukhin > > ср, 5 февр. 2020 г. в 17:44, Denis Magda : >> >> Alex, did you have a chance to review Saikat’s changes related to the >> extensions repository organization and release approach? >> >> Denis >> >> On Wednesday, February 5, 2020, Nikolay Izhikov wrote: >> >>> Hello, Igniters. >>> >>> We don't have a release process for newly created ignite-extensions >>> modules. >>> I want to release two modules: >>> >>> * ignite-spring-boot-autoconfigure >>> * ignite-client-spring-boot-autoconfigure >>> >>> Let's discuss it. >>> >>> Any objections to it? >>> What should be done before release? >>> >> >> >> -- >> - >> Denis
Re: [DISCUSS] Public API deprecation rules
Thanks, updated the justification: The vote we are going to have is reduced to two choices so far: * Never deprecate the old APIs unless the new APIs are stable and released without @IgniteExperimental. The old APIs javadoc may be updated with a reference to new APIs to encourage users to evaluate new APIs. The deprecation and new API release may happen simultaneously if the new API is not marked with @IgniteExperimental or the annotation is removed in the same release. * Allow to deprecate the old APIs even when new APIs are marked with @IgniteExperimental to explicitly notify users that old APIs will be removed in the next major release AND new APIs are available As for the @IgniteExperimental precise meaning - I do not agree that the experimental essence is related to the API only. As you suggested in the separate email, we are going to mark MVCC as experimental because it is in beta meaning that there are currently known scenarios when the new feature breaks or simply is not implemented. Yet the API for MVCC cache is quite stable and adds nothing more than a separate cache mode. Not sure if this changes the vote, though.
Re: Mark MVCC with @IgniteExperimental
Agree, let's mark MVCC experimental. Stephen, the annotation serves as an additional documentation-style marker. For now there are no compile-time warnings when the API is used. чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < stephen.darling...@gridgain.com>: > Yes! I’ve already seen people try to use this without awareness that it’s > not production ready. > > What happens with the annotation, incidentally? Is it just in the > documentation or do you get a compile-time warning? > > > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > > > Hello, Igniters. > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > We explicitly note users that MVCC has beta status, for now [1] > > > >> Beta version of Transactional SQL and MVCC > >> In Ignite v2.7, Transactional SQL and MVCC are released as beta > versions to allow users to experiment and share feedback. > >> This version of Transactional SQL and MVCC should not be considered for > production. > > > > > > [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > >
Re: Mark MVCC with @IgniteExperimental
Hello, I think the right answer is YES. It should be marked with the new annotation @IgniteExperimental. Thanks, S. чт, 6 февр. 2020 г. в 14:35, Stephen Darlington < stephen.darling...@gridgain.com>: > Yes! I’ve already seen people try to use this without awareness that it’s > not production ready. > > What happens with the annotation, incidentally? Is it just in the > documentation or do you get a compile-time warning? > > > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > > > Hello, Igniters. > > > > Should we mark MVCC feature with the new @IgniteExperimental? > > > > We explicitly note users that MVCC has beta status, for now [1] > > > >> Beta version of Transactional SQL and MVCC > >> In Ignite v2.7, Transactional SQL and MVCC are released as beta > versions to allow users to experiment and share feedback. > >> This version of Transactional SQL and MVCC should not be considered for > production. > > > > > > [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control > > > > > > >
Re: Question: different setting for nodes.
Nikolay, This sounds like a philosophical question :) By rigid I mean inability to change some properties without whole cluster restart, and I do think it's bad. I have a good example in mind - the number of threads running rebalance. Originally, we added a validation check for this configuration, so it was impossible to change it without cluster restart. There was a user who was running a cluster; the volume of data grew and afterwards they needed to increase rebalance speed. But they could not do this. Once we removed the validation and added proper handling of different rebalance threads number, the user managed to restart nodes one-by-one, wait for rebalance and keep the cluster up and running. So I think the general direction should be handle automatically as much as possible and make behavior consistent, rather than introduce additional restrictions.
Re: Mark MVCC with @IgniteExperimental
Yes! I’ve already seen people try to use this without awareness that it’s not production ready. What happens with the annotation, incidentally? Is it just in the documentation or do you get a compile-time warning? > On 6 Feb 2020, at 11:32, Nikolay Izhikov wrote: > > Hello, Igniters. > > Should we mark MVCC feature with the new @IgniteExperimental? > > We explicitly note users that MVCC has beta status, for now [1] > >> Beta version of Transactional SQL and MVCC >> In Ignite v2.7, Transactional SQL and MVCC are released as beta versions to >> allow users to experiment and share feedback. >> This version of Transactional SQL and MVCC should not be considered for >> production. > > > [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control > >
Mark MVCC with @IgniteExperimental
Hello, Igniters. Should we mark MVCC feature with the new @IgniteExperimental? We explicitly note users that MVCC has beta status, for now [1] > Beta version of Transactional SQL and MVCC > In Ignite v2.7, Transactional SQL and MVCC are released as beta versions to > allow users to experiment and share feedback. > This version of Transactional SQL and MVCC should not be considered for > production. [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control
Re: Excessive backups performance suggestion.
>> most users do not want to lose data by default. But some ready to sacrifice. I know at least one case when "pure speed without insurance" required when the ML model is studying. On Thu, Feb 6, 2020 at 1:39 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > I'm quite sure most users do not want to lose data by default. > > It's not wise to recommend such thing even with explicit warning. > > чт, 6 февр. 2020 г. в 13:30, Anton Vinogradov : > > > The statement seems to be correct for cases when we'd like to perform > fast > > loading and computation and not afraid of data loss. > > Performance boost always means we lose some guarantee. > > Let's just update the proposal with the explicit warning that reducing > > backups to zero may lead to data loss on any node failure. > > > > On Thu, Feb 6, 2020 at 1:09 PM Alexei Scherbakov < > > alexey.scherbak...@gmail.com> wrote: > > > > > Definitely wrong recommendation. > > > Please file a ticket for removal. > > > > > > чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky > > > > >: > > > > > > > > > > > Igniters, i found that suggesting zero backup for performance > > > improvements > > > > sounds like malicious and further problems suggestion. May be it`s > time > > > to > > > > remove it ? > > > > > > > > Example: > > > > [07:59:27] Performance suggestions for grid (fix if possible) > > > > [07:59:27] To disable, set > > -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > > > > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0) > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > > > > > -- > > Best regards, > Alexei Scherbakov >
Re: Excessive backups performance suggestion.
I'm quite sure most users do not want to lose data by default. It's not wise to recommend such thing even with explicit warning. чт, 6 февр. 2020 г. в 13:30, Anton Vinogradov : > The statement seems to be correct for cases when we'd like to perform fast > loading and computation and not afraid of data loss. > Performance boost always means we lose some guarantee. > Let's just update the proposal with the explicit warning that reducing > backups to zero may lead to data loss on any node failure. > > On Thu, Feb 6, 2020 at 1:09 PM Alexei Scherbakov < > alexey.scherbak...@gmail.com> wrote: > > > Definitely wrong recommendation. > > Please file a ticket for removal. > > > > чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky > > >: > > > > > > > > Igniters, i found that suggesting zero backup for performance > > improvements > > > sounds like malicious and further problems suggestion. May be it`s time > > to > > > remove it ? > > > > > > Example: > > > [07:59:27] Performance suggestions for grid (fix if possible) > > > [07:59:27] To disable, set > -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > > > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0) > > > > > > > > > > > > > > -- > > > > Best regards, > > Alexei Scherbakov > > > -- Best regards, Alexei Scherbakov
Permissions
Hello! I want to be a new community member so I ask you to give me contributors permissions https://issues.apache.org/jira/secure/ViewProfile.jspa?name=artsiom_panko
Re: [DISCUSS] Public API deprecation rules
Hello, I have two notes > * Allow to deprecate the old APIs even when new APIs are marked with > @IgniteExperimental to explicitly notify users that the new APIs are available I think the main reason is not «notify users that new APIs are available», but «Notify users that old APIs will be removed in the next major release AND new APIs are available» > @IgniteExperimental [3] which marks APIs which are yet not stable, dangerous, > partially implemented or are allowed to be changed in a future. It seems too wide definition of «experimental API» I think we can’t release «dangerous» API. For me, `@IgniteExperimental` should mark public APIs that can be slightly improved or changed in the next minor release. > 6 февр. 2020 г., в 13:23, Andrey Gura написал(а): > > One comment about first choice. > > The formulation has one issue from my point of view: Never deprecate > the old APIs unless the new APIs are stable *and released* without > @IgniteExperimental. > > Actually we can introduce new stable API and deprecate old API at the > same time without releasing of new API. For example > IgniteCache.withAsync() method was deprecated and new methods > xxxAsync() were added at the same time as stable alternative. In this > case we didn't collect users' feedback because negative users' > feedback about withAsync() was a driver for this change. > > On Thu, Feb 6, 2020 at 6:26 AM Denis Magda wrote: >> >> The justification and choices look crystal clear to me. >> >> - >> Denis >> >> >> On Wed, Feb 5, 2020 at 1:36 PM Alexey Goncharuk >> wrote: >>> >>> Igniters, >>> >>> We would like to add some formal requirements to the API deprecation >>> process. So far the API deprecation was more like intuitive-driven which >>> recently led to some disagreement and delaying the AI 2.8 release [1][2]. >>> >>> During the discussion [1] we agreed to add an >>> annotation @IgniteExperimental [3] which marks APIs which are yet not >>> stable, dangerous, partially implemented or are allowed to be changed in a >>> future. The reason to release such APIs is to expose the APIs to users and >>> collect feedback on the usability and possible bugs. >>> >>> The argument we did not manage to resolve is whether we are allowed to >>> deprecate the old APIs _before_ the new ones get stable and get released >>> without @IgniteExperimental annotation. We decided to resolve the >>> uncertainty with a vote. >>> >>> The vote we are going to have is reduced to two choices so far: >>> * Never deprecate the old APIs unless the new APIs are stable and released >>> without @IgniteExperimental. The old APIs javadoc may be updated with a >>> reference to new APIs to encourage users to evaluate new APIs >>> * Allow to deprecate the old APIs even when new APIs are marked >>> with @IgniteExperimental to explicitly notify users that the new APIs are >>> available >>> >>> Nikolay, Andrey, please let us know if we should correct the choices >>> articulation or add another option for the vote. >>> >>> --AG >>> >>> [1] >>> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html >>> [2] >>> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html >>> [3] https://issues.apache.org/jira/browse/IGNITE-12559
Re: [DISCUSS] Public API deprecation rules
Agree, I was implying this option. Corrected: The vote we are going to have is reduced to two choices so far: * Never deprecate the old APIs unless the new APIs are stable and released without @IgniteExperimental. The old APIs javadoc may be updated with a reference to new APIs to encourage users to evaluate new APIs. The deprecation and new API release may happen simultaneously if the new API is not marked with @IgniteExperimental or the annotation is removed in the same release. * Allow to deprecate the old APIs even when new APIs are marked with @IgniteExperimental to explicitly notify users that the new APIs are available Nikolay, any comments from your side?
Re: Excessive backups performance suggestion.
The statement seems to be correct for cases when we'd like to perform fast loading and computation and not afraid of data loss. Performance boost always means we lose some guarantee. Let's just update the proposal with the explicit warning that reducing backups to zero may lead to data loss on any node failure. On Thu, Feb 6, 2020 at 1:09 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > Definitely wrong recommendation. > Please file a ticket for removal. > > чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky >: > > > > > Igniters, i found that suggesting zero backup for performance > improvements > > sounds like malicious and further problems suggestion. May be it`s time > to > > remove it ? > > > > Example: > > [07:59:27] Performance suggestions for grid (fix if possible) > > [07:59:27] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0) > > > > > > > > -- > > Best regards, > Alexei Scherbakov >
Re: Internal classes are exposed in public API
The discussion thread is started: http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Public-API-deprecation-rules-td45647.html ср, 5 февр. 2020 г. в 12:43, Alexey Goncharuk : > Sorry for the rush, I think I got it - it's right in the voting process > [1]. This would be a vote on a procedural issue of how to handle code > deprecation. > > I'll start the discussion soon, let's agree on the alternatives and then > start the vote. > > [1] https://www.apache.org/foundation/voting.html >
Re: [DISCUSS] Public API deprecation rules
One comment about first choice. The formulation has one issue from my point of view: Never deprecate the old APIs unless the new APIs are stable *and released* without @IgniteExperimental. Actually we can introduce new stable API and deprecate old API at the same time without releasing of new API. For example IgniteCache.withAsync() method was deprecated and new methods xxxAsync() were added at the same time as stable alternative. In this case we didn't collect users' feedback because negative users' feedback about withAsync() was a driver for this change. On Thu, Feb 6, 2020 at 6:26 AM Denis Magda wrote: > > The justification and choices look crystal clear to me. > > - > Denis > > > On Wed, Feb 5, 2020 at 1:36 PM Alexey Goncharuk wrote: >> >> Igniters, >> >> We would like to add some formal requirements to the API deprecation >> process. So far the API deprecation was more like intuitive-driven which >> recently led to some disagreement and delaying the AI 2.8 release [1][2]. >> >> During the discussion [1] we agreed to add an >> annotation @IgniteExperimental [3] which marks APIs which are yet not >> stable, dangerous, partially implemented or are allowed to be changed in a >> future. The reason to release such APIs is to expose the APIs to users and >> collect feedback on the usability and possible bugs. >> >> The argument we did not manage to resolve is whether we are allowed to >> deprecate the old APIs _before_ the new ones get stable and get released >> without @IgniteExperimental annotation. We decided to resolve the >> uncertainty with a vote. >> >> The vote we are going to have is reduced to two choices so far: >> * Never deprecate the old APIs unless the new APIs are stable and released >> without @IgniteExperimental. The old APIs javadoc may be updated with a >> reference to new APIs to encourage users to evaluate new APIs >> * Allow to deprecate the old APIs even when new APIs are marked >> with @IgniteExperimental to explicitly notify users that the new APIs are >> available >> >> Nikolay, Andrey, please let us know if we should correct the choices >> articulation or add another option for the vote. >> >> --AG >> >> [1] >> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html >> [2] >> http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html >> [3] https://issues.apache.org/jira/browse/IGNITE-12559
Re: Excessive backups performance suggestion.
Definitely wrong recommendation. Please file a ticket for removal. чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky : > > Igniters, i found that suggesting zero backup for performance improvements > sounds like malicious and further problems suggestion. May be it`s time to > remove it ? > > Example: > [07:59:27] Performance suggestions for grid (fix if possible) > [07:59:27] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0) > > -- Best regards, Alexei Scherbakov
Re: Question: different setting for nodes.
Hello, Alexey. Sorry, my question is not related to the current discussion > this makes the cluster configuration rigid; Is it a bad thing to make cluster node configuration rigid? > 6 февр. 2020 г., в 12:20, Alexey Goncharuk > написал(а): > > Vladimir, > > In current implementation the flag does not change if data is kept or > cleared from memory - after deactivation the internal structures are > cleared and only data region is kept allocated. If you add a validation for > this flag, you forbid users to do a rolling cluster restart and > enable/disable this behavior without full cluster restart, which itself > means total data loss in case of an in-memory cluster. > > Same stands for all other system properties. Unless the property is > absolutely required to be the same on all nodes, there is no need to > validate, this makes the cluster configuration rigid; such properties may > be changed only with the whole cluster downtime. > > ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov : > >> Folks, >> >> >> I think it's a common problem for such system properties (which may be >> different on different nodes and which may lead to unpredictable >> cluster behaviour). The same mentioned by Ivan here [1]. >> >> Is it better to validate (somehow) all system properties, for >> instance, started with IGNITE_ prefix, on all nodes the same way as we >> do it for IgniteConfiguration? >> >> [1] >> https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207 >> >> On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov wrote: >>> >>> Vladimir. >>> >>> This looks like a bug to me. >>> >>> Can you, please, prepare the simple reproducer for this issue and it’s >> consequences? >>> >>> 5 февр. 2020 г., в 17:08, Vladimir Steshin >> написал(а): Hi, folks. I recently found that one node might be started with flag reuse-memory-on-deactivate [1] set off while another node might be configured with the flag enabled. This ability hinders prediction of cluster behavior on deactivation/activation. One node keeps data in >> memory, other doesn’t. Should we check this flag is set equally on all nodes? Is this an >> issue? We might prevent node start if it has different setting. [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE >>> >>
Excessive backups performance suggestion.
Igniters, i found that suggesting zero backup for performance improvements sounds like malicious and further problems suggestion. May be it`s time to remove it ? Example: [07:59:27] Performance suggestions for grid (fix if possible) [07:59:27] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [07:59:27] ^-- Decrease number of backups (set 'backups' to 0)
Re: Question: different setting for nodes.
Vladimir, In current implementation the flag does not change if data is kept or cleared from memory - after deactivation the internal structures are cleared and only data region is kept allocated. If you add a validation for this flag, you forbid users to do a rolling cluster restart and enable/disable this behavior without full cluster restart, which itself means total data loss in case of an in-memory cluster. Same stands for all other system properties. Unless the property is absolutely required to be the same on all nodes, there is no need to validate, this makes the cluster configuration rigid; such properties may be changed only with the whole cluster downtime. ср, 5 февр. 2020 г. в 18:40, Maxim Muzafarov : > Folks, > > > I think it's a common problem for such system properties (which may be > different on different nodes and which may lead to unpredictable > cluster behaviour). The same mentioned by Ivan here [1]. > > Is it better to validate (somehow) all system properties, for > instance, started with IGNITE_ prefix, on all nodes the same way as we > do it for IgniteConfiguration? > > [1] > https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207 > > On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov wrote: > > > > Vladimir. > > > > This looks like a bug to me. > > > > Can you, please, prepare the simple reproducer for this issue and it’s > consequences? > > > > > > > 5 февр. 2020 г., в 17:08, Vladimir Steshin > написал(а): > > > > > > Hi, folks. > > > > > > > > > > > > I recently found that one node might be started with flag > > > reuse-memory-on-deactivate [1] set off while another node might be > > > configured with the flag enabled. This ability hinders prediction of > > > cluster behavior on deactivation/activation. One node keeps data in > memory, > > > other doesn’t. > > > > > > Should we check this flag is set equally on all nodes? Is this an > issue? We > > > might prevent node start if it has different setting. > > > > > > > > > > > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > >
Re: [REVIEW REQUEST] IGNITE-12630 Remove developers sections from parent pom.xml
Looks good to me. On Thu, Feb 6, 2020 at 11:45 AM Ivan Pavlukhin wrote: > Igniters, > > I raised a PR for a ticket [1] removing section from > parent pom.xml. I described the motivation in the ticket. Shortly, > this section has a little meaning today and even worse is misleading. > > Please review. > > [1] https://issues.apache.org/jira/browse/IGNITE-12630 > > Best regards, > Ivan Pavlukhin >
[REVIEW REQUEST] IGNITE-12630 Remove developers sections from parent pom.xml
Igniters, I raised a PR for a ticket [1] removing section from parent pom.xml. I described the motivation in the ticket. Shortly, this section has a little meaning today and even worse is misleading. Please review. [1] https://issues.apache.org/jira/browse/IGNITE-12630 Best regards, Ivan Pavlukhin
[jira] [Created] (IGNITE-12631) Incorrect rewriting wal record type in marshalled mode during iteration
Anton Kalashnikov created IGNITE-12631:
--
Summary: Incorrect rewriting wal record type in marshalled mode
during iteration
Key: IGNITE-12631
URL: https://issues.apache.org/jira/browse/IGNITE-12631
Project: Ignite
Issue Type: Bug
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov
The fail happens on iteration over wal record which was written under
marshalled mode in case when RecordType#ordinal != RecordType#index
{noformat}
[16:46:58,800][SEVERE][pitr-ctx-exec-#399][GridRecoveryProcessor] Fail scan wal
log for recovery localNodeConstId=node_1_1
class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at
position: 45905 size: -1
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.handleRecordException(AbstractWalRecordsIterator.java:292)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.handleRecordException(FileWriteAheadLogManager.java:3302)
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:258)
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advance(AbstractWalRecordsIterator.java:154)
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:123)
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:52)
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.nextX(GridCloseableIteratorAdapter.java:41)
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.next(GridIteratorAdapter.java:35)
... 7 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read WAL
record at position: 45905 size: -1
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:394)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer.readRecord(RecordV2Serializer.java:235)
at
org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:243)
... 12 more
Caused by: java.io.IOException: Unknown record type: null, expected pointer
[idx=2, offset=45905]
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer$2.readWithHeaders(RecordV2Serializer.java:122)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:373)
... 14 more
Suppressed: class
org.apache.ignite.internal.processors.cache.persistence.wal.crc.IgniteDataIntegrityViolationException:
val: 1445348818 writtenCrc: 374280888
at
org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput$Crc32CheckingFileInput.close(FileInput.java:106)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:380)
... 14 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12630) Remove developers sections from parent pom.xml
Ivan Pavlukhin created IGNITE-12630: --- Summary: Remove developers sections from parent pom.xml Key: IGNITE-12630 URL: https://issues.apache.org/jira/browse/IGNITE-12630 Project: Ignite Issue Type: Bug Reporter: Ivan Pavlukhin Assignee: Ivan Pavlukhin Fix For: 2.9 We have a section in parent/pom.xml. Listed info in fact is outdated and it is not practical and useful to enumerate everyone in this section. Moreover the proper position here is that the project is developed by Apache Ignite Community. Let's remove this section. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSSION] ignite-extension - ignite-spring-boot-autoconfigurer release.
Nikolay, Thank you for driving it! It is great to establish this process in practice earlier because it seems that we need to release a Flink integration soon because Ignite 2.8 is going to be released without that integration bundled. As usual we need to decide about versioning and a correspondence to Ignite versions. As we are going to have a separate release cycle I can imagine an independent versioning scheme with a range of supported Ignite versions. Also I think it is a good idea to update ignite-extensions README on GitHub and provide a link to relevant TC jobs. Best regards, Ivan Pavlukhin ср, 5 февр. 2020 г. в 17:44, Denis Magda : > > Alex, did you have a chance to review Saikat’s changes related to the > extensions repository organization and release approach? > > Denis > > On Wednesday, February 5, 2020, Nikolay Izhikov wrote: > > > Hello, Igniters. > > > > We don't have a release process for newly created ignite-extensions > > modules. > > I want to release two modules: > > > > * ignite-spring-boot-autoconfigure > > * ignite-client-spring-boot-autoconfigure > > > > Let's discuss it. > > > > Any objections to it? > > What should be done before release? > > > > > -- > - > Denis
