Re: PDS suites fail with exit code 137

2020-07-27 Thread Ivan Pavlukhin
Ivan B., Good news, thank you! 2020-07-27 10:28 GMT+03:00, Ivan Bessonov : > Hi Ivan P., > > I configured it for both PDS (Indexing) and PDS 4 (was asked by Nikita > Tolstunov). It totally worked, not a single 137 since then. > Occasional 130 will be fixed in [1], it has a different problem behin

Re: PDS suites fail with exit code 137

2020-07-27 Thread Ivan Bessonov
Hi Ivan P., I configured it for both PDS (Indexing) and PDS 4 (was asked by Nikita Tolstunov). It totally worked, not a single 137 since then. Occasional 130 will be fixed in [1], it has a different problem behind it. Now I'm trying to find someone who knows TC configuration better and will be ab

Re: PDS suites fail with exit code 137

2020-07-26 Thread Ivan Pavlukhin
Ivan B., I noticed that you were able to configure environment variables for PDS (Indexing). Do field experiments show that the suggested approach fixes the problem? Interesting stuff with jemalloc. It might be useful to file a ticket. 2020-07-23 16:07 GMT+03:00, Ivan Daschinsky : >> >> About "j

Re: PDS suites fail with exit code 137

2020-07-23 Thread Ivan Daschinsky
> > About "jemalloc" - it's also an option, but it also requires reconfiguring > suites on > TC, maybe in a more complicated way. It requires additional installation, > right? > Can we stick to the solution that I already tested or should we update TC > agents? :) Yes, if you want to use jemalloc

Re: PDS suites fail with exit code 137

2020-07-23 Thread Ivan Bessonov
> > glibc allocator uses arenas for minimize contention between threads I understand it the same way. I did testing with running of Indexing suite locally and periodically executing "pmap ", it showed that the number of 64mb arenas grows constantly and never shrinks. By the middle of the suite th

Re: PDS suites fail with exit code 137

2020-07-23 Thread Ivan Daschinsky
AFAIK, glibc allocator uses arenas for minimize contention between threads when they trying to access or free preallocated bit of memory. But seems that we use -XX:+AlwaysPreTouch, so heap is allocated and committed at start time. We allocate memory for durable memory in one thread. So I think ther

Re: PDS suites fail with exit code 137

2020-07-23 Thread Ivan Bessonov
Hello Ivan, It feels like the problem is more about new starting threads rather than the allocation of offheap regions. Plus I'd like to see results soon, your proposal is a major change for Ignite that can't be implemented fast enough. Anyway, I think this makes sense, considering that one day U

Re: PDS suites fail with exit code 137

2020-07-23 Thread Ivan Daschinsky
Ivan, I think that we should use mmap/munmap to allocate huge chunks of memory. I've experimented with JNA and invoke mmap/munmap with it and it works fine. May be we can create module (similar to direct-io) that use mmap/munap on platforms, that support them and fallback to Unsafe if not? чт, 23

PDS suites fail with exit code 137

2020-07-23 Thread Ivan Bessonov
Hello Igniters, I'd like to discuss the current issue with "out of memory" fails on TeamCity. Particularly suites [1] and [2], they have quite a lot of "Exit code 137" failures. I investigated the "PDS (Indexing)" suite under [3]. There's another similar issue as well: [4]. I came to the conclusi