Re: [WANTED A NEW RELEASE MANAGER] Apache Ignite 3.0.0 beta 1 RELEASE
Hi, I’ll be happy to do this. I’ll need help from a PMC to do the steps requiring PMC permissions but I can do most of the work. Any PMC who is ready to support me in this? Thanks, Stan > On 13 Oct 2022, at 15:54, Andrey Gura wrote: > > Igniters, > > Due to personal reasons I need to take a pause so we need a new > release manager for the Apache Ignite 3 beta 1 release. > > The best option is a PMC member. Committer is also a good option, but > it will need some help from PMC members. > > Please feel free to submit your candidacy in this thread.
Re: IEP-70: Async Continuation Executor
Pavel, This is a great work, fully agree with the overall idea and approach. However, I have some reservations about the API. We sure do have a lot of async stuff in the system, and I would suggest to stick to the usual design - create a separate thread pool, add a single property to control the size of the pool. Alternatively, we may consider using public pool for that. May I ask if there is an example use case which doesn’t work with public pool? For .NET, agree that we should follow the rules and APIs of the platform, so the behavior might slightly differ. Thanks, Stan > On 24 Mar 2021, at 09:52, Pavel Tupitsyn wrote: > > Igniters, since there are no more comments and/or review feedback, > I'm going to merge the changes by the end of the week. > >> On Mon, Mar 22, 2021 at 10:37 PM Pavel Tupitsyn >> wrote: >> >> Ready for review: >> https://github.com/apache/ignite/pull/8870 >> >> On Sun, Mar 21, 2021 at 8:09 PM Pavel Tupitsyn >> wrote: >> >>> Simple benchmark added - see JmhCacheAsyncListenBenchmark in the PR. >>> There is a 6-8% drop (1 client, 2 servers, 1 machine, int key/val). >>> I expect this difference to become barely observable on real-world >>> workloads. >>> >>> On Thu, Mar 18, 2021 at 12:35 PM Pavel Tupitsyn >>> wrote: >>> Denis, For a reproducer, please see CacheAsyncContinuationExecutorTest.java in the linked PoC [1] [1] https://github.com/apache/ignite/pull/8870/files#diff-c788c12013622093df07d8f628a6e8c5fb0c15007f0787f2d459dbb3e377fc5aR54 On Thu, Mar 18, 2021 at 1:58 AM Raymond Wilson < raymond_wil...@trimble.com> wrote: > We implemented the ContinueWith() suggestion from Pavel, and it works > well > so far in testing, though we do not have data to support if there is or > is > not a performance penalty in our use case.. > > To lend another vote to the 'Don't do continuations on the striped > thread > pool' line of thinking: Deadlocking is an issue as is starvation. In > some > ways starvation is more insidious because by the time things stop > working > the cause and effect distance may be large. > > I appreciate the documentation does make statements about not performing > cache operations in a continuation due to deadlock possibilities, but > that > statement does not reveal why this is an issue. It is less a case of a > async cache operation being followed by some other cache operation (an > immediate issue), and more a general case of the continuation of > application logic using a striped pool thread in a way that might mean > that > thread is never given back - it's now just a piece of the application > infrastructure until some other application activity schedules away from > that thread (eg: by ContinueWith or some other async operation in the > application code that releases the thread). To be fair, beyond > structures > like ContinueWith(), it is not obvious how that continuation thread > should > be handed back. This will be the same behaviour for dedicated > continuation > pools, but with far less risk in the absence of ContinueWith() > constructs. > > In the .Net world this is becoming more of an issue as fewer .Net use > cases > outside of UI bother with synchronization contexts by default. > > Raymond. > > > On Thu, Mar 18, 2021 at 9:56 AM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > >> Hi Denis, >> >> I think Pavel's main point is that behavior is unpredictable. For > example, >> AFAIK, putAsync can be executed in the main thread instead of the > striped >> pool thread if the operation is local. The listener can also be > executed in >> the main thread - this happens if the future is completed prior to > listener >> invocation (this is actually quite possible in the unit test > environment >> causing the test to pass). Finally, I'm pretty sure there are many > cases >> when a deadlock does not occur right away, but instead it will reveal >> itself under high load due to thread starvation. The latter type of > issues >> is the most dangerous because they are often reproduced only in > production. >> Finally, there are performance considerations as well - cache > operations >> and listeners share the same fixed-sized pool which can have negative >> effects. >> >> I'm OK with the change. Although, it might be better to introduce a > new >> fixed-sized pool instead of ForkJoinPool for listeners, simply > because this >> is the approach taken throughout the project. But this is up to a > debate. >> >> -Val >> >> On Wed, Mar 17, 2021 at 11:31 AM Denis Garus > wrote: >> >>> Pavel, >>> I tried this: >>> >>> @Test >>> public void
RE: [DISCUSSION] [IGNITE-11141] The new java method to create a cache from a template
Sergey, Ed,
On method naming/deprecation/etc.
I would actually like the new method to work for both templates and regular
caches.
For templates it would return a copy of the template.
For existing caches it would return a copy of the cache configuration.
In other words, it would be a shortcut for
`new
CacheConfiguration(ignite.cache("foo").configuration(CacheConfiguration.class))`
I wouldn't expect it to be widely used but I think this adds some uniformity to
the behavior.
Ilya,
On asterisk usage.
If you have a cache config "foo*" in your static cache configurations
then you will be able to get that configuration via
`ignite.cacheConfiguration("foo")` (no asterisk).
Asterisk is being dropped when the template is created. This is how it works
now, no changes here.
All,
Regarding reshuffling all this in 3.0.
I would
- change method name addCacheConfiguration to addCacheTemplate
- would add a new property `CacheConfiguration[] cacheTemplates` to put
templates to - instead of putting them
into `cacheConfigutations` with an asterisk
But that's a different topic, I think.
Thanks,
Stan
> -Original Message-
> From: Ilya Kasnacheev
> Sent: Thursday, February 14, 2019 3:32 PM
> To: dev@ignite.apache.org
> Subject: Re: [DISCUSSION] [IGNITE-11141] The new java method to create a
> cache from a template
>
> Hello!
>
> What about existing convention of using asterisk to mark templates? How
> does it integerate with this one?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 14 февр. 2019 г. в 13:04, Sergey Moldachev
> :
>
> > Sounds good, I agree with naming and marking *addCacheConfiguration*
> method
> > as deprecated with replacing on *addCacheConfigurationTemplate*.
> >
> > Stanislav, could you please look at this?
> >
> > Thank you,
> > Sergey
> >
> > чт, 14 февр. 2019 г. в 01:49, Eduard Shangareev <
> > eduard.shangar...@gmail.com
> > >:
> >
> > > CacheConfiguration cfg = ignite.cacheConfiguration("myTemplate");
> > > cfg.setName("myCacheFromTemplate");
> > > ignite.createCache(cfg);
> > >
> > > Ok, I got it. We already have addCacheConfiguration and
> > cacheConfiguration
> > > should be a getter-like counterpart for it.
> > >
> > > So, I would suggest deprecating this addCacheConfiguration method and
> add
> > > new one Ignite.addCacheTemplate and its counterpart
> getCacheTemplate.
> > > Because cacheConfiguration looks very weird, I would expect that it
> > should
> > > return cache configuration for any given cache name.
> > >
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Feb 14, 2019 at 12:01 AM Sergey Moldachev <
> > > sergeymoldac...@gmail.com>
> > > wrote:
> > >
> > > > Hi, Eduard, thank you for your reply!
> > > >
> > > > You can find example and full description in the Jira task
> > > > . Also you can
> > find
> > > > simple implementation in comments.
> > > >
> > > > Thank you,
> > > > Sergey!
> > > >
> > > > + update subject (fixed a typo)
> > > >
> > > > ср, 13 февр. 2019 г. в 19:55, Eduard Shangareev <
> > > > eduard.shangar...@gmail.com
> > > > >:
> > > >
> > > > > Hi, Sergey!
> > > > >
> > > > > Could you give some example of how it is supposed to use?
> > > > >
> > > > > On Wed, Feb 13, 2019 at 6:02 PM Sergey Moldachev <
> > > > > sergeymoldac...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, Igniters!
> > > > > >
> > > > > > I want to add the new java method *cacheConfiguration(String
> > > > cacheName)*
> > > > > in
> > > > > > to *Ignite* core interface as part of the task
> > > > > > https://issues.apache.org/jira/browse/IGNITE-11141.
> > > > > >
> > > > > > I'll be glad to see the comments about this feature.
> > > > > >
> > > > > > Thank you,
> > > > > > Sergey
> > > > > >
> > > > >
> > > >
> > >
> >
RE: [DISCUSSION] Fixed test order to reduce flakyness
HI Dmitry, > It is not about hiding. If a problematic test affects other test is will > continue to > affect. The main point here is only about the test, which will be affected. > With unpredictable order, testA may break testB, testC, testD. The issue is that with a fixed order testA may not break any tests (e.g. because it will become last). Also, tests may not only break but "fix" each other, i.e. testA may make a should-be-failing testB pass. > > But for predictive & fixed order test affected by a failure of testA will be > always testC, and B will not be considered flaky because of the > randomized nature of execution. It will help for building at least good > statistics in the TC Bot. I agree that random test order may sound too scary and that file-system dependent order is no good because it's like a fixed one, but varies across platforms. In my experience the best way to tackle execution order of tests is to vary direct and reverse fixed order (say, alphabetical). It gives enough difference to make sure order-dependent tests fail sometimes, and, most importantly, it gives a way to catch them (you can tell that a problem is related to order when a test only fails on direct/reverse runs). I see that surefire already has in a form of runOrder=hourly. I suggest that if we change to something, we change to that. Stan > -Original Message- > From: Dmitriy Pavlov > Sent: Friday, February 8, 2019 10:19 PM > To: dev@ignite.apache.org > Subject: Re: [DISCUSSION] Fixed test order to reduce flakyness > > I mean changing parameter > http://maven.apache.org/surefire/maven-surefire-plugin/test- > mojo.html#runOrder > which > is fileSystem by default. > > It is not about hiding. If a problematic test affects other test is will > continue to > affect. The main point here is only about the test, which will be affected. > With unpredictable order, testA may break testB, testC, testD. > > But for predictive & fixed order test affected by a failure of testA will be > always testC, and B will not be considered flaky because of the > randomized nature of execution. It will help for building at least good > statistics in the TC Bot. > > ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev > >: > > > Dmitriy, > > > > Please, clarify the idea. > > What do we want to achieve by this? Making tests more stable by hiding > > some issues in our tests/codebase? > > > > > > On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван > wrote: > > > > > Dmitriy, > > > > > > Sounds like a good idea to me. Problems related to tests isolation > > > are very common in practice. And things can be complicated when > > > order varies. > > > > > > But by the way does the order of Ignite tests vary today? Junit 4 > > > javadocs claims something about "default deterministic order" [1]. > > > > > > [1] > > https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html > > > > > > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov : > > > > > > > > Dear Ignite Developers, > > > > > > > > The original idea came from our recent habr.ru post related to > > > > Apache Ignite TeamCity Bot (for Russian native speakers, you can > > > > read an > > > original > > > > > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 > > > > ) > > > > > > > > It is a known phenomenon when tests have an influence on each other. > > The > > > > simplest case when Ignite Native persistence is used, and not > > > > properly cleared after a test run. This can make some test failed > afterward. > > > > > > > > So, what if we will set predictable, for example, alphabetical > > > > tests execution order > > > > (maven-surefire-plugin/runOrder/alphabetical). This may have the > > > > following effect: the set of tests failed because of being > > > > affected by the previous run will be constant, will be exactly the same > each run. > > > > > > > > At some point, when we stabilize flaky tests enough, we may select > > random > > > > order, but for now, this solution seems valid to me. > > > > > > > > What do you think? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > >
RE: Binary Serialization of Enums By Name
Hi Stuart,
I can see the value of storing enums by name. One thing is the problem you’ve
described. The other is that they become more readable when the enum class is
not present which is the case sometimes.
However, I find having two built-in ways to serialize enums quite complicated.
I’m sure it will lead to more issues in corner cases.
So the only thing I would look into is possible migration from ordinal-based to
name-based enum serialization in 3.0. But that’s 3.0.
To solve your issue I would rather go with explicitly specified ordinals like
public enum Foo {
ONE(1), TWO(2), THREE(3)
}
Stan
From: Stuart Macdonald
Sent: Thursday, February 7, 2019 2:52 PM
To: dev ; u...@ignite.apache.org
Subject: Re: Binary Serialization of Enums By Name
Adding dev list.
In the absence of any way of modifying binary serialization of enums, would the
dev community consider accepting a pull request? Two options I would consider:
1. A custom “enum serialiser” on the BinaryTypeConfiguration and an associated
enum serialization interface.
2. A flag on BinaryConfiguration oto specify serialization of enums by name
rather than ordinal.
Any other ideas would be appreciated.
Stuart.
On Mon, 4 Feb 2019 at 16:35, Stuart Macdonald mailto:stu...@stuwee.org> > wrote:
Hi Mike,
Thanks for the response. I can’t see how that’s possible with the current
Binarylizable interface as enums are immutable, there’s no nullary constructor
for an enum and we can’t update its name after construction. I gave it a shot
to see what would happen but the read/write methods weren’t being called at all.
Any other suggestions would be much appreciated.
Stuart.
On 4 Feb 2019, at 16:22, Michael Cherkasov mailto:michael.cherka...@gmail.com> > wrote:
Hi Stuart,
I think you can use Binarylizable interface, you can implement your one
serialization for your enum.
Thanks,
Mike.
пн, 4 февр. 2019 г. в 15:12, Stuart Macdonald mailto:stu...@stuwee.org> >:
Igniters,
I have some cache objects which contain enum fields, which when persisted
through Ignite binary persistence are persisted using their enum ordinal
position. However these enums are often modified whereby new values are
inserted in the middle of the existing enum values, which breaks
deserialization by ordinal. Does anyone know of a way to have Ignite serialize
enums by name (ie. standard java enum serialization), or to allow for custom
serialization routines for enums?
Many thanks,
Stuart.
