Re: [Distributed SQL] Do we have a plan to implement QuadTree index?

2018-08-03 Thread Denis Magda
Alexey, are you working on some new ML/DL APIs/algorithms? Please elaborate what you'd like to add to Ignite. -- Denis On Wed, Aug 1, 2018 at 3:10 PM Pavel Kovalenko wrote: > Hello Alexey, > > It's not so difficult to implement new type of indexing of data, but if you > want to reach

Re: can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

2018-08-03 Thread Denis Magda
Hi Deepa, What is your use case? There might be a solution. -- Denis On Fri, Aug 3, 2018 at 1:44 AM Deepa Kolwalkar wrote: > Hi > > While searching for a solution on how to have DB updates to reflect > automatically in Ignite Cache, I came across this post dt Mar-2018 which > states that : >

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Denis Magda
Anton, Peter, thanks a lot! We all owe you a jar of beer and chocolate (?) ;) Send me an invoice and I'll pay for it :) -- Denis On Fri, Aug 3, 2018 at 7:32 AM Petr Ivanov wrote: > Yeap. When PR will be ready, I’ll start updating build configurations. > > > > On 3 Aug 2018, at 17:28, Anton

Re: Ignite as distributed file storage

2018-08-03 Thread Denis Magda
Dmitriy, I would suggest us not limiting blobs use case to a dedicated cache. If to look at other databases they usually have BLOB/LONGBLOB/etc. as a type meaning that users mix simple and BLOB types in the same tables. Should we start with Ignite SQL adding blobs through its APIs? -- Denis On

Re: Code inspection

2018-08-03 Thread Dmitriy Pavlov
Ideally, I should not asking for people to solve something for me. I just want this idea did not disappear from our radar. пт, 3 авг. 2018 г. в 23:47, Dmitriy Setrakyan : > On Fri, Aug 3, 2018 at 7:49 AM, Dmitriy Pavlov > wrote: > > > > > I understand it is not so Apache-way from my side to

Re: Page replacement policy improvements (when persistent is enabled)

2018-08-03 Thread Dmitriy Setrakyan
Vladimir, Are we only counting timestamp of the last access? In that case, it would create a problem. We should also count number of times a page has been touched within a certain time frame, e.g. last hour or so. In this case, index pages would not be evicted as they get touched the most. I

Re: Ignite as distributed file storage

2018-08-03 Thread Dmitriy Setrakyan
Pavel, Not everything that gets put in Ignite has a class, and not everything can be annotated. For example, what about byte[] or a binary object? I would prefer a separate cache with specific purpose of storing blobs. It will be easier for users to understand and configure. It can also have a

Re: Code inspection

2018-08-03 Thread Dmitriy Setrakyan
On Fri, Aug 3, 2018 at 7:49 AM, Dmitriy Pavlov wrote: > > I understand it is not so Apache-way from my side to ask volunteers to do > some things (instead of contributing it by myself). Dmitriy, I am not sure why you feel this is not the Apache way. No one can do everything themselves. You

Re: IgniteSparkSession Should Copy State on cloneSession()

2018-08-03 Thread Nikolay Izhikov
Hello, Stuart. Looks like a bug. Please, go ahead with fixing it. В Пт, 03/08/2018 в 12:43 +0100, Stuart Macdonald пишет: > Hello Igniters, > > The IgniteSparkSession class extends SparkSession and overrides the > cloneSession() method. The contract for cloneSession() explicitly states > that

[GitHub] ignite pull request #4484: IGNITE-9186: Omit datastreamer per-thread buffers...

2018-08-03 Thread AMashenkov
GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/4484 IGNITE-9186: Omit datastreamer per-thread buffers for already batched. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (IGNITE-9186) Datastreamer may ignores perNodeBuffer size unexpectedly.

2018-08-03 Thread Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-9186: Summary: Datastreamer may ignores perNodeBuffer size unexpectedly. Key: IGNITE-9186 URL: https://issues.apache.org/jira/browse/IGNITE-9186 Project: Ignite

[MTCGA]: new failures in builds [1570331] needs to be handled

2018-08-03 Thread dpavlov . tasks
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *New test failure in master ExamplesTest.TestRemoteNodes(BinaryModeExample)

[MTCGA]: new failures in builds [1578455] needs to be handled

2018-08-03 Thread dpavlov . tasks
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *New Critical Failure in master Platform .NET (Long Running)

Re: Adding experimental support for Intel Optane DC Persistent Memory

2018-08-03 Thread Dmitriy Pavlov
Hi Mulugeta, I appologise, I've missed that license is already there. So I guess it is not standard open-source license, it is seems it is not listed in https://www.apache.org/legal/resolved.html#category-a So there can be legal concern related to including this lib as dependency into Apache

[jira] [Created] (IGNITE-9185) Collect and check update counters visited during WAL rebalance

2018-08-03 Thread Pavel Kovalenko (JIRA)
Pavel Kovalenko created IGNITE-9185: --- Summary: Collect and check update counters visited during WAL rebalance Key: IGNITE-9185 URL: https://issues.apache.org/jira/browse/IGNITE-9185 Project: Ignite

[GitHub] ignite pull request #4447: IGNITE-9043 let values of any type be written to ...

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4447 ---

[GitHub] ignite pull request #4461: IGNITE-9134: ZookeeperDiscoverySpiTest#testLargeU...

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4461 ---

[GitHub] ignite pull request #4479: IGNITE-9170: fix resVer for local caches

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4479 ---

[GitHub] ignite pull request #4472: IGNITE-9148

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4472 ---

Re: Code inspection

2018-08-03 Thread Dmitriy Pavlov
Hi Igniters, I understand it is not so Apache-way from my side to ask volunteers to do some things (instead of contributing it by myself). But I still interested - if we need code inspection or not, - and who would like to spend odd hour and sett up some regular/manual scans. Sincerely, Dmitriy

[GitHub] ignite pull request #4471: IGNITE-9159 Basic 2 TC configuration is halted by...

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4471 ---

Re: Exchange stucks while node restoring state from WAL

2018-08-03 Thread Pavel Kovalenko
Hello Maxim, 1) Yes, Discovery Manager is starting after GridCacheProcessor, which starts GridCacheDatabaseSharedManager which invokes readMetastorage on start. 2) Before we complete the local join future, we create and add Exchange future on local node join to ExchangeManager. So, when local

Re: [BENCHMARK REQUEST] IGNITE-8446

2018-08-03 Thread Ilya Suntsov
Anton, I've run benchmarks with the following parameters: - 4 servers, 8 clients - primary/full sync - in-memory Benchmarks show approximately the same numbers. Primary sync: ignite-8446 ignite-8446-master delta tx-optim-repRead-put-get 36119 36241 -0.34% tx-pessim-repRead-put-get

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Petr Ivanov
Yeap. When PR will be ready, I’ll start updating build configurations. > On 3 Aug 2018, at 17:28, Anton Vinogradov wrote: > > I can perform proper refactoring by myself. > > Peter, could you assist me with TC in that case? > > пт, 3 авг. 2018 г. в 17:17, Dmitriy Setrakyan : > >> Agree with

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Anton Vinogradov
I can perform proper refactoring by myself. Peter, could you assist me with TC in that case? пт, 3 авг. 2018 г. в 17:17, Dmitriy Setrakyan : > Agree with Denis. Let's do the simple refactoring first, and then more > complicated one in phase 2. > > On Fri, Aug 3, 2018 at 6:47 AM, Denis Magda

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Dmitriy Setrakyan
Agree with Denis. Let's do the simple refactoring first, and then more complicated one in phase 2. On Fri, Aug 3, 2018 at 6:47 AM, Denis Magda wrote: > > > > Thus, my suggestion is: > > — find and update all hardcoded “fabric” usage on TC, so that they work > > both with or without fabric in

[jira] [Created] (IGNITE-9184) Cluster hangs during concurrent node restart and continues query registration

2018-08-03 Thread Mikhail Cherkasov (JIRA)
Mikhail Cherkasov created IGNITE-9184: - Summary: Cluster hangs during concurrent node restart and continues query registration Key: IGNITE-9184 URL: https://issues.apache.org/jira/browse/IGNITE-9184

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Denis Magda
> > Thus, my suggestion is: > — find and update all hardcoded “fabric” usage on TC, so that they work > both with or without fabric in name of binaries > — use current implementation — review and merge to master > — plan full suffix refactoring (as Anton suggests) on the next iterations > with

[GitHub] ignite pull request #4483: IGNITE-9149 Add proper handling of UUID column in...

2018-08-03 Thread ivandasch
GitHub user ivandasch opened a pull request: https://github.com/apache/ignite/pull/4483 IGNITE-9149 Add proper handling of UUID column in DDL statements. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite

[jira] [Created] (IGNITE-9183) Proper handling UUID columns, that are added by DDL.

2018-08-03 Thread Ivan Daschinskiy (JIRA)
Ivan Daschinskiy created IGNITE-9183: Summary: Proper handling UUID columns, that are added by DDL. Key: IGNITE-9183 URL: https://issues.apache.org/jira/browse/IGNITE-9183 Project: Ignite

[jira] [Created] (IGNITE-9182) Propagate lazy flag to synthetic SELECT used by first step of DELETE

2018-08-03 Thread Stanislav Lukyanov (JIRA)
Stanislav Lukyanov created IGNITE-9182: -- Summary: Propagate lazy flag to synthetic SELECT used by first step of DELETE Key: IGNITE-9182 URL: https://issues.apache.org/jira/browse/IGNITE-9182

Re: [BENCHMARK REQUEST] IGNITE-8446

2018-08-03 Thread Anton Vinogradov
- Benchmark should check how many tx can be completed in second (please use same as you use to check the release) - in-memory - default - same you use at pre-release check. пт, 3 авг. 2018 г. в 15:38, Ilya Suntsov : > Anton, > > Please provide more details: > >- Benchmark class name >-

Re: [BENCHMARK REQUEST] IGNITE-8446

2018-08-03 Thread Ilya Suntsov
Anton, Please provide more details: - Benchmark class name - PDS/in-memory - sync mode - cluster size ? 2018-08-03 13:52 GMT+03:00 Dmitriy Pavlov : > Hi Anton, > > Thank you for bringing this question here. Actually I'm not right person > who can do it, but hopefully Ilya can

[jira] [Created] (IGNITE-9181) Continuous query with remote filter factory doesn't let nodes join

2018-08-03 Thread Denis Mekhanikov (JIRA)
Denis Mekhanikov created IGNITE-9181: Summary: Continuous query with remote filter factory doesn't let nodes join Key: IGNITE-9181 URL: https://issues.apache.org/jira/browse/IGNITE-9181 Project:

[GitHub] ignite pull request #4482: IGNITE-9068 : Avoided potential deadlock when ser...

2018-08-03 Thread ilantukh
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/4482 IGNITE-9068 : Avoided potential deadlock when service is deployed while node is stopping. You can merge this pull request into a Git repository by running: $ git pull

Creation of dynamic cache

2018-08-03 Thread Denis Garus
Hello, Igniters! If an error occurred during of creation dynamic cache, the CacheAffinitySharedManager#processCacheStartRequests method will try to rollback cache start routine. ``` try { if (startCache) { cctx.cache().prepareCacheStart(req.startCacheConfiguration(),

IgniteSparkSession Should Copy State on cloneSession()

2018-08-03 Thread Stuart Macdonald
Hello Igniters, The IgniteSparkSession class extends SparkSession and overrides the cloneSession() method. The contract for cloneSession() explicitly states that it should clone all state (ie. the sharedState and sessionState fields), however the IgniteSparkSession implementation doesn't clone

Re: Ignite as distributed file storage

2018-08-03 Thread Alexey Kuznetsov
Pavel, Will this internal storage mounted on all nodes hosts? Otherwise we will have a huge traffic on rebalance of blobs. On Fri, Aug 3, 2018 at 5:21 PM Pavel Kovalenko wrote: > Dmitriy, > > I think we don't need a separate implementation of cache like BLOB cache. > Instead of it, a user can

[jira] [Created] (IGNITE-9180) IgniteSparkSession Should Copy State on cloneSession()

2018-08-03 Thread Stuart Macdonald (JIRA)
Stuart Macdonald created IGNITE-9180: Summary: IgniteSparkSession Should Copy State on cloneSession() Key: IGNITE-9180 URL: https://issues.apache.org/jira/browse/IGNITE-9180 Project: Ignite

[GitHub] ignite pull request #4436: IGNITE-9064: Decision tree optimization

2018-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4436 ---

Re: [BENCHMARK REQUEST] IGNITE-8446

2018-08-03 Thread Dmitriy Pavlov
Hi Anton, Thank you for bringing this question here. Actually I'm not right person who can do it, but hopefully Ilya can assist. Hi Ilya, could you please run required benchmark? Sincerely, Dmitriy Pavlov пт, 3 авг. 2018 г. в 13:33, Anton Vinogradov : > Dmitry, > > AFAIK, you're able to

[BENCHMARK REQUEST] IGNITE-8446

2018-08-03 Thread Anton Vinogradov
Dmitry, AFAIK, you're able to perform benchmarking of critical code. Could you please check https://issues.apache.org/jira/browse/IGNITE-8446? We have to check https://github.com/apache/ignite/tree/ignite-8446 vs https://github.com/apache/ignite/tree/ignite-8446-master Only one benchmark is

Re: Ignite as distributed file storage

2018-08-03 Thread Pavel Kovalenko
Dmitriy, I think we don't need a separate implementation of cache like BLOB cache. Instead of it, a user can mark value class or value class field with the special annotation "@BLOB". During cache put, marshaller will place a special placeholder on such fields, write byte[] array payload of a

Mass TC RunAll: 8th iteration

2018-08-03 Thread Dmitriy Pavlov
Hi Igniters, I'm going to trigger mass run all to be executed during weekend. I hope you don't mind. I will trigger build 17:00 UTC, 20:00 MSK, 10:00 PST. Should you have urgent PR to be checked, feel free to reorder it to top. Sincerely, Dmitriy Pavlov

Re: IP finder in tests

2018-08-03 Thread Yakov Zhdanov
Well, I tend to agree. Can you try applying it on some suite and share the results in terms of run time decrease and decreasing number of random failures due to improper tests stop or cleanup? --Yakov 2018-08-02 16:37 GMT+03:00 Denis Mekhanikov : > Yakov, > > Almost every test in the project

[jira] [Created] (IGNITE-9179) Web Console: Split VisorBaselineTask for two parts in order to simplify tooling

2018-08-03 Thread Alexey Kuznetsov (JIRA)
Alexey Kuznetsov created IGNITE-9179: Summary: Web Console: Split VisorBaselineTask for two parts in order to simplify tooling Key: IGNITE-9179 URL: https://issues.apache.org/jira/browse/IGNITE-9179

Re: [MTCGA]: new failures in builds [1575775] needs to be handled

2018-08-03 Thread Yakov Zhdanov
Maxim, did we have .net tests failing before merge? If yes, how come we merged it? --Yakov 2018-08-02 18:03 GMT+03:00 Maxim Muzafarov : > Folks, > > Seems like rebalancing changes leads us to hanging TestRebalance() test for > .NET. > Created issue [1], I will try to investigate it. > > [1]

Re: can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

2018-08-03 Thread Dmitriy Pavlov
Hi, if this question is not related to some contribution than, probably, better place to ask is u...@ignite.apache.org mailing list. Sincerely, Dmitriy Pavlov пт, 3 авг. 2018 г. в 11:44, Deepa Kolwalkar : > Hi > > While searching for a solution on how to have DB updates to reflect >

Re: Page replacement policy improvements (when persistent is enabled)

2018-08-03 Thread Dmitriy Pavlov
Hi Vladimir, I really feel that page replacement approach can be improved. Currently I don't think that page nature will give us much, because usage frequency can be independent to page type. I also noticed a couple of tickets were done by Ilya Kasnacheev and Eugeniy Stanilovskly, which were

[jira] [Created] (IGNITE-9178) Partition lost event are not triggered if multiple nodes left cluster

2018-08-03 Thread Pavel Vinokurov (JIRA)
Pavel Vinokurov created IGNITE-9178: --- Summary: Partition lost event are not triggered if multiple nodes left cluster Key: IGNITE-9178 URL: https://issues.apache.org/jira/browse/IGNITE-9178 Project:

can Ignite Cache automatically detect changes in multiple Underlying DBs and stay in sync with them

2018-08-03 Thread Deepa Kolwalkar
Hi While searching for a solution on how to have DB updates to reflect automatically in Ignite Cache, I came across this post dt Mar-2018 which states that : == Ignite does not provide such integration out of the box, however there a commercial offering from GridGain for that:

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Petr Ivanov
Dmitriy, I cannot forecast estimates for this task as it dependents on many factors: — I will be able to start researching the Anton’s implementation suggestion not earlier than the beginning of September — I am not acquainted with assembly configuration well, it may take some considerable

[GitHub] ignite pull request #4481: IGNITE-9165 Close streams in core module

2018-08-03 Thread zzzadruga
GitHub user zzzadruga opened a pull request: https://github.com/apache/ignite/pull/4481 IGNITE-9165 Close streams in core module You can merge this pull request into a Git repository by running: $ git pull https://github.com/zzzadruga/ignite IGNITE-9165 Alternatively you

[GitHub] ignite pull request #4480: IGNITE-8640 earlier validation of cache config

2018-08-03 Thread dgarus
GitHub user dgarus opened a pull request: https://github.com/apache/ignite/pull/4480 IGNITE-8640 earlier validation of cache config You can merge this pull request into a Git repository by running: $ git pull https://github.com/dgarus/ignite ignte-8640 Alternatively you can

Re: Exchange stucks while node restoring state from WAL

2018-08-03 Thread Nikolay Izhikov
Hello, Maxim. > 1) Is it correct that readMetastore() happens after node starts> but before > including node into the ring?> I think yes. You can have some kind of metainformation required on node join. > 5) Does in our final solution for new joined node readMetastore> and > restoreMemory

Page replacement policy improvements (when persistent is enabled)

2018-08-03 Thread Vladimir Ozerov
Igniters, I heard some complaints about our page replacement algorithm that index pages could be evicted from memory too often. I reviewed our current implementation and looks like we have choosen very simple approach with eviction of random pages, without taking in count their nature (data vs

Re: Removing "fabric" from Ignite binary package name

2018-08-03 Thread Vladimir Ozerov
Folks, Can you please explain the problem with TC and artifacts? Can we just rename final artifact at the end of a build phase just before signing, and leave the rest TC infrastructure as is? On Fri, Aug 3, 2018 at 12:28 AM Dmitriy Setrakyan wrote: > Anton, Petr, > > Thanks for your readiness

Exchange stucks while node restoring state from WAL

2018-08-03 Thread Maxim Muzafarov
Hi Igniters, I'm working on bug [1] and have some questions about the final implementation. Probably, I've already found answers on some of them but I want to be sure. Please, help me to clarify details. The key problem here is that we are reading WAL and restoring memory state of new joined