[jira] [Created] (IGNITE-10453) Web console: Implement possibility to configure disk page compression.
Vasiliy Sisko created IGNITE-10453: -- Summary: Web console: Implement possibility to configure disk page compression. Key: IGNITE-10453 URL: https://issues.apache.org/jira/browse/IGNITE-10453 Project: Ignite Issue Type: Task Components: wizards Reporter: Vasiliy Sisko Assignee: Vasiliy Sisko Implemented possibility to configure disk page compression that is implemented in IGNITE-10330 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Change code style inspections to use red mark for those used at Teamcity build checks (IGNITE-10450)
Hi Oleg, I am not ultimately sure that it will not do any harm. Red error highlights are designed to attract attention very well. I can imagine that I will be distracted by every unused import. While usually I rely on "Optimize imports" enabled in the commit dialog in IDEA. But I think that we can try it. ср, 28 нояб. 2018 г. в 21:12, Dmitriy Pavlov : > > Sure, I agree to let the discussion run its course and give it a couple of > days so people have a chance to think and chime in. > > I just wanted to show I'm ++1 here and probably we can employ > Commit-Than-Review later > > > ср, 28 нояб. 2018 г. в 20:40, oignatenko : > > > Hi Dmitry, > > > > When we had preliminary discussion of this with Maxim we both were inclined > > to post it here and let it hang for a while to let dev list folks discuss > > this idea in more details for just in case if we missed some usability > > implications. > > > > Though now that you mentioned it I figured that proposed change is low risk > > and easy to rollback, meaning we can do it the other way round: just merge > > it now and keep in mind an option to revert if further discussion here > > shows > > that this way is wrong for some reason. > > > > If you prefer that we pick this way, changing priorities for TC inspections > > could even be done as a part of another ticket involving this config file, > > IGNITE-10422 - you can probably discuss with Maxim if he would be > > comfortable with that (last time I checked he planned to do implementation > > there). > > > > regards, Oleg > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > -- Best regards, Ivan Pavlukhin
[jira] [Created] (IGNITE-10452) Inconsistent state of caches if a node stops in the process of running transactions
Roman Guseinov created IGNITE-10452: --- Summary: Inconsistent state of caches if a node stops in the process of running transactions Key: IGNITE-10452 URL: https://issues.apache.org/jira/browse/IGNITE-10452 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.6 Reporter: Roman Guseinov Attachments: NonAcidTxReproducer.java It seems it happens if several caches are used in a transaction. And there are some caches with enabled CacheStore and other ones with disabled. If all caches have CacheStore (or no one has) then the issue doesn't occur. Reproducer is attached (tested on master branch). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)
Hi Alex, I've just started implement of the view. Thanks for the your efforts! ср, 28 нояб. 2018 г. в 19:00, Alex Plehanov : > Yuriy, > > If you have plans to implement running queries view in the nearest future, > I already have implemented draft for local node queries some time ago [1]. > Maybe it will help to implement a view for whole cluster queries. > > [1]: > > https://github.com/alex-plekhanov/ignite/commit/6231668646a2b0f848373eb4e9dc38d127603e43 > > > ср, 28 нояб. 2018 г. в 17:34, Vladimir Ozerov : > > > Denis > > > > I would wait for running queries view first. > > > > ср, 28 нояб. 2018 г. в 1:57, Denis Magda : > > > > > Vladimir, > > > > > > Please see inline > > > > > > On Mon, Nov 19, 2018 at 8:23 AM Vladimir Ozerov > > > wrote: > > > > > > > Denis, > > > > > > > > I partially agree with you. But there are several problem with syntax > > > > proposed by you: > > > > 1) This is harder to implement technically - more parsing logic to > > > > implement. Ok, this is our internal problem, users do not care about > it > > > > 2) User will have to consult to docs in any case > > > > > > > > > > Two of these are not a big deal. We just need to invest more time for > > > development and during the design phase so that people need to consult > > the > > > docs rarely. > > > > > > > > > > 3) "nodeId" is not really node ID. For Ignite users node ID is UUID. > In > > > our > > > > case this is node order, and we intentionally avoided any naming > here. > > > > > > > > > > Let's use a more loose name such as "node". > > > > > > > > > > Query is just identified by a string, no more than that > > > > 4) Proposed syntax is more verbose and open ways for misuse. E.g. > what > > is > > > > "KILL QUERY WHERE queryId=1234"? > > > > > > > > I am not 100% satisfied with both variants, but the first one looks > > > simpler > > > > to me. Remember, that user will not guess query ID. Instead, he will > > get > > > > the list of running queries with some other syntax. What we need to > > > > understand for now is how this syntax will look like. I think that we > > > > should implement getting list of running queries, and only then start > > > > working on cancellation. > > > > > > > > > > That's a good point. Syntax of both running and killing queires > commands > > > should be tightly coupled. We're going to name a column if running > > queries > > > IDs somehow anyway and that name might be resued in the WHERE clause of > > > KILL. > > > > > > Should we discuss the syntax in a separate thread? > > > > > > -- > > > Denis > > > > > > > > > > > Vladimir. > > > > > > > > > > > > On Mon, Nov 19, 2018 at 7:02 PM Denis Mekhanikov < > > dmekhani...@gmail.com> > > > > wrote: > > > > > > > > > Guys, > > > > > > > > > > Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me. > > > > > I'm going to look up in documentation, which parameter goes first > in > > > this > > > > > query every time I use it. > > > > > I like the syntax, that Igor suggested more. > > > > > Will it be better if we make *nodeId* and *queryId *named > properties? > > > > > > > > > > Something like this: > > > > > KILL QUERY WHERE nodeId=25 and queryId=1234 > > > > > > > > > > Denis > > > > > > > > > > пт, 16 нояб. 2018 г. в 14:12, Юрий : > > > > > > > > > > > I fully agree with last sentences and can start to implement this > > > part. > > > > > > > > > > > > Guys, thanks for your productive participate at discussion. > > > > > > > > > > > > пт, 16 нояб. 2018 г. в 2:53, Denis Magda : > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > Thanks, make perfect sense to me. > > > > > > > > > > > > > > > > > > > > > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov < > > > > voze...@gridgain.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Denis, > > > > > > > > > > > > > > > > The idea is that QueryDetailMetrics will be exposed through > > > > separate > > > > > > > > "historical" SQL view in addition to current API. So we are > on > > > the > > > > > same > > > > > > > > page here. > > > > > > > > > > > > > > > > As far as query ID I do not see any easy way to operate on a > > > single > > > > > > > integer > > > > > > > > value (even 64bit). This is distributed system - we do not > want > > > to > > > > > have > > > > > > > > coordination between nodes to get query ID. And coordination > is > > > the > > > > > > only > > > > > > > > possible way to get sexy "long". Instead, I would propose to > > form > > > > ID > > > > > > from > > > > > > > > node order and query counter within node. This will be (int, > > > long) > > > > > > pair. > > > > > > > > For use convenience we may convert it to a single string, > e.g. > > > > > > > > "[node_order].[query_counter]". Then the syntax would be: > > > > > > > > > > > > > > > > KILL QUERY '25.1234'; // Kill query 1234 on node 25 > > > > > > > > KILL QUERY '25.*; // Kill all queries on the node 25 > > > > > > > > > > > > > > > > Makes sense? > > > > > > > > > > > > > > > >
[jira] [Created] (IGNITE-10451) .NET: Persistence does not work with custom affinity function
Pavel Tupitsyn created IGNITE-10451:
---
Summary: .NET: Persistence does not work with custom affinity
function
Key: IGNITE-10451
URL: https://issues.apache.org/jira/browse/IGNITE-10451
Project: Ignite
Issue Type: Bug
Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
To reproduce: assign custom affinity function in
{{PersistenceTest.TestCacheDataSurvivesNodeRestart}}.
As a result, node restart fails with the following exception:
{code}
Apache.Ignite.Core.Common.IgniteException : An error occurred during cache
configuration loading from file
[file=C:\Users\tps0\AppData\Local\Temp\Ignite_ihxso0zq.tw0\Store\node00-263cfb5e-ec70-4378-8cbb-62b6fcc8043b\cache-persistentCache\cache_data.dat]
> Apache.Ignite.Core.Common.JavaException : class
org.apache.ignite.IgniteException: An error occurred during cache configuration
loading from file
[file=C:\Users\tps0\AppData\Local\Temp\Ignite_ihxso0zq.tw0\Store\node00-263cfb5e-ec70-4378-8cbb-62b6fcc8043b\cache-persistentCache\cache_data.dat]
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1027)
at
org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:48)
at
org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:74)
Caused by: class org.apache.ignite.IgniteCheckedException: An error occurred
during cache configuration loading from file
[file=C:\Users\tps0\AppData\Local\Temp\Ignite_ihxso0zq.tw0\Store\node00-263cfb5e-ec70-4378-8cbb-62b6fcc8043b\cache-persistentCache\cache_data.dat]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.readCacheData(FilePageStoreManager.java:902)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.readCacheConfigurations(FilePageStoreManager.java:844)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.addCacheOnJoinFromConfig(GridCacheProcessor.java:891)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.restoreCacheConfigurations(GridCacheProcessor.java:756)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1300(GridCacheProcessor.java:204)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.onReadyForRead(GridCacheProcessor.java:5456)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:412)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:724)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4473)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1047)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656)
at
org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
... 1 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
deserialize object with given class loader:
sun.misc.Launcher$AppClassLoader@18b4aac2
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:147)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.readCacheData(FilePageStoreManager.java:898)
... 15 more
Caused by: java.lang.IllegalArgumentException: Ignite instance name thread
local must be set or this method should be accessed under
org.apache.ignite.thread.IgniteThread
at
org.apache.ignite.internal.IgnitionEx.localIgnite(IgnitionEx.java:1413)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.threadLocalContext(GridBinaryMarshaller.java:398)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.readExternal(BinaryObjectImpl.java:695)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
Re: Case sensitive indexes question.
Hi Zhenya, Answering your questions: 1) Yes, this is expected 2) This is standard rule applicable for almost all vendors and all SQL commands - object name without quotes is normalized to some case (upper or lower), object name in qoutes is left as is 3) Hard to say, need to experiment with it 4) We don't Some vendors allow it (MySQL, Postgres), may be some other's don't. We cannot restrict their usage in AI 2.x as it may break existing applications. Neither I think we should restrict it - nothing is broken. IMO what we do need is to inform user that he is doing something strange and let him decide what to do. This could be a warning in the log, special "performance suggestions" SQL view, whatever. Vladimir. On Wed, Nov 28, 2018 at 8:57 PM Zhenya wrote: > Igniters, i found that ignite allow to create multiple case sensitive > indexes with equal fields collection, i.e. no exception and warn here: > > CREATE INDEX \"title_idx\" ON books (title); > CREATE INDEX \"tiTLE_IDX\" ON books (title); > > 1. in this case will be created two different index structures. > 2. documentation [1] not clarify that quotation usage will create > different ones and quotation absence will create index name in upper > registry. > 3. what index, query planner would be use? > 4. and main question: why do we need this functional? > > i found that other vendors not allow such indexes [2] > > if we can`t change this in 2.x version due to some backward compatibility > reasons, plz show them, i have no clue why we can`t change it in near > release versions. > > [1] https://apacheignite-sql.readme.io/docs/create-index > [2] > > https://oracle-base.com/articles/12c/multiple-indexes-on-same-set-of-columns-12cr1 > > thanks! >
Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files
Maxim, Regarding MVCC - this is essentially a copy-on-write approach. New entry is created on every update. They are cleaned asynchronously by dedicated threads (aka "vacuum"). I looked at the document you mentioned, thank you for pointing to it. But it doesn't answer all questions around existing design, and what I am trying to do is to get how deep do we understand current problems. It is very true that various subsystems, such as buffer managers, WALs, supporting sctructures, etc. incur very serious overhead. And when it comes to heavy operations implementors typically seek for a way to bypass as much components as possible, taking in count that different shortcuts lead to different types of side effects. And IMO our very important goal for now is to create space of possible improvements, and estimate their costs, risks and applicability for product's configuration space. Let me claridy several questions about current rebalance implementation, as I am not a big expert here. 1) Is it correct that supplier sends only one message for every demand message? If yes, then streaming should improve network utilization a lot. 2) Is it correct that for user caches we process supply messages in a system pool? Did we consider moving it to striped pool? Because if all operations on a single partition is ordered, we may apply a number of critical optimizations - bypassing page cache and checkpointer for new entries, batched index updates, batched free list updates, etc. 3) Seems that WAL should no longer be a problem for us [1]. What are exact conditions when it could be disabled on supplier side? 4) Most important - have we tried to profile plain single-threaded rebalance without concurrent write load? We need to have clear understanding on where time is spent - supplier/demander, cpu/network/disk, etc. Some Java tracing code should help. And one question regarding proposed implementation - how are we going to handle secondary indexes? [1] https://issues.apache.org/jira/browse/IGNITE-8017 On Wed, Nov 28, 2018 at 6:43 PM Maxim Muzafarov wrote: > Eduard, > > Thank you very much for the discussion! > > Your algorithm looks much better for me too and easier to implement. > I'll update appropriate process points on IEP page of the proposed > rebalance procedure. > On Tue, 27 Nov 2018 at 18:52, Eduard Shangareev > wrote: > > > > So, after some discussion, I could describe another approach on how to > > build consistent partition on the fly. > > > > 1. We make a checkpoint, fix the size of the partition in OffheapManager. > > 2. After checkpoint finish, we start sending partition file (without any > > lock) to the receiver from 0 to fixed size. > > 3. Next checkpoints if they detect that they would override some pages of > > transferring file should write the previous state of a page to a > dedicated > > file. > > So, we would have a list of pages written 1 by 1, page id is written in > the > > page itself so we could determine page index. Let's name it log. > > 4. When transfer finished checkpointer would stop updating log-file. Now > we > > are ready to send it to the receiver. > > 5. On receiver side we start merging the dirty partition file with log > > (updating it with pages from log-file). > > > > So, an advantage of this method: > > - checkpoint-thread work couldn't increase more than twice; > > - checkpoint-threads shouldn't wait for anything; > > - in best case, we receive partition without any extra effort. > > > > > > On Mon, Nov 26, 2018 at 8:54 PM Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Maxim, > > > > > > I have looked through your algorithm of reading partition consistently. > > > And I have some questions/comments. > > > > > > 1. The algorithm requires heavy synchronization between > checkpoint-thread > > > and new-approach-rebalance-threads, > > > because you need strong guarantees to not start writing or reading to > > > chunk which was updated or started reading by the counterpart. > > > > > > 2. Also, if we have started transferring this chunk in original > partition > > > couldn't be updated by checkpoint-threads. They should wait for > transfer > > > finishing. > > > > > > 3. If sending is slow and partition is updated then in worst case > > > checkpoint-threads would create the whole copy of the partition. > > > > > > So, what we have: > > > -on every page write checkpoint-thread should synchronize with > > > new-approach-rebalance-threads; > > > -checkpoint-thread should do extra-work, sometimes this could be as > huge > > > as copying the whole partition. > > > > > > > > > On Fri, Nov 23, 2018 at 2:55 PM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > > wrote: > > > > > >> Hello! > > >> > > >> This proposal will also happily break my compression-with-dictionary > patch > > >> since it relies currently on only having local dictionaries. > > >> > > >> However, when you have compressed data, maybe speed boost is even > greater > > >> with your approach. > >
[GitHub] ignite pull request #5529: Ignite 2.4.14
GitHub user aealeksandrov opened a pull request: https://github.com/apache/ignite/pull/5529 Ignite 2.4.14 Created for TS run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.14 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5529.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5529 commit 7cfc7377e1a0d08d56ac4b36995b8bf0d91609b9 Author: Stanislav Lukyanov Date: 2018-04-17T14:15:08Z IGNITE-8210 Fixed custom event handling for baseline topology change - Fixes #3814. Signed-off-by: Alexey Goncharuk (cherry picked from commit d79c640) commit 83b9ffd1307c17435c1aae81f42480d90404f8a2 Author: ezhuravl Date: 2018-04-17T15:41:56Z IGNITE-6113 Changed mistake in version for partition demand version commit afc5fc1789d75573f71b40c5e241484c7a578197 Author: ezhuravl Date: 2018-04-17T15:41:56Z IGNITE-6113 Changed mistake in version for partition demand version (cherry picked from commit 83b9ffd) commit 25f6a2013aa584559623410b7a96951f79fb00ff Author: Ivan Daschinskiy Date: 2018-04-17T16:55:42Z IGNITE-8021 Delete cache config files when cache is destroyed - Fixes #3697. Signed-off-by: Alexey Goncharuk (cherry picked from commit 2edcb22fbb566981097733af6470ed6dde8e786b) commit 5461dd64ee15f02be7934b33d0ca92130aa70512 Author: Ilya Kasnacheev Date: 2018-04-17T17:04:28Z IGNITE-2766 Fix .net test. commit 9f5b27fae9ac57ae5b256cb8593dfe587b4accb8 Author: oleg-ostanin Date: 2018-04-17T17:58:53Z IGNITE-8274 sqlline.sh script uses JAVA_HOME now Signed-off-by: Andrey Gura (cherry picked from commit c3ff274) commit 640167f2c9384fddd69e6244b615e4974bfe2b50 Author: Maxim Muzafarov Date: 2018-04-18T09:20:13Z IGNITE-8301 testReconnectCacheDestroyedAndCreated should excpect recreated client cache. Cherry-picked from 56be24b9dfc14023bacaab63f40e0504b317eda3 commit 89b8426a2a113b6893a2295044d6dc0e94015a94 Author: Alexey Kuznetsov Date: 2018-04-18T11:49:12Z ignite-2.4.4 Fixed default node version. commit 048c21a3cc7d00a1c5951137f3747904e00405ea Author: Alexey Kuznetsov Date: 2018-04-19T07:14:51Z IGNITE-8298 Web Console: Fixed tables UI issues. (cherry picked from commit f3848a2) commit 18a3ba0f6dc07729f78a24b345dbfc1588cdb4c2 Author: Dmitriy Shabalin Date: 2018-04-19T08:16:18Z IGNITE-8298 Web Console: Fixed loader under Safari. (cherry picked from commit 0897309) commit 0499793d49d5e48d5fdec97bbb8c2ac609e5056e Author: Ivan Daschinskiy Date: 2018-04-19T12:25:23Z IGNITE-8021 Fixed tests - Fixes #3864. Signed-off-by: Alexey Goncharuk (cherry picked from commit 8fc1824) commit 8b21e0b36d7d035ff52bcff067f002140f4b8b97 Author: Alexey Kuznetsov Date: 2018-03-23T10:53:15Z IGNITE-7119 Web Agent: Implemented support for comma-separated list of node URIs. (cherry picked from commit ee0f4c0) commit a9f63143059fc489342cadc0c89d7d8fd389fdff Author: Denis Mekhanikov Date: 2018-04-20T14:11:36Z ignite-8205 Clear list of local services in GridServiceProcessor#onKernalStop Signed-off-by: Andrey Gura (cherry picked from commit fbe24f8e3b0d9016a69670ca2bc50766865adf38) commit 2aa4d60df18e57f28814675cf37298ba952035b7 Author: Denis Mekhanikov Date: 2018-04-20T15:41:06Z IGNITE-8134 Subscribe to system cache events on nodes outside BLT Signed-off-by: Andrey Gura (cherry picked from commit c82277eb4e48f95dfec8cb0206c019820a765432) commit ef140ce1102c37295fe9c52d4fcc52b7bdd2bb09 Author: Alexey Kuznetsov Date: 2018-04-23T08:44:09Z IGNITE-8298 Web Console: Fixed tables UI issues. commit 561950f4afc37a078eefc54664f56bdff6d2dcfd Author: Anton Kurbanov Date: 2018-04-21T18:23:21Z IGNITE-8154 - Add an ability to provide ExceptionListener to JmsStreamer - Fixes #3828 Signed-off-by: Valentin Kulichenko commit 1dbd6970fd2ce611c0cbbfa9256b08a934fc8666 Author: Anton Kurbanov Date: 2018-04-23T09:24:50Z Merge branch 'ignite-2.4-master' of https://github.com/gridgain/apache-ignite into ignite-2.4-master commit cafbff336761c5464cb60b68b0f7193d5c998d9f Author: Andrey V. Mashenkov Date: 2018-04-16T17:43:36Z IGNITE-7972 Fixed NPE in TTL manager on unwindEvicts. - Fixes #3810. Signed-off-by: dpavlov (cherry picked from commit 737933e) commit 16fa0132be0cce8e2af2566fd7ad06a741b5fee0 Author: Andrey V. Mashenkov Date: 2018-02-07T15:25:25Z IGNITE-7508: Fix contention on system property access in GridKernalContextImpl::isDaemon(). This closes #3468. (cherry picked from commit d2b41a0) commit 996e3f5b39746777eecad73bc303838fe76121c2 Author: tledkov-gridgain Date:
Re: Change code style inspections to use red mark for those used at Teamcity build checks (IGNITE-10450)
Sure, I agree to let the discussion run its course and give it a couple of days so people have a chance to think and chime in. I just wanted to show I'm ++1 here and probably we can employ Commit-Than-Review later ср, 28 нояб. 2018 г. в 20:40, oignatenko : > Hi Dmitry, > > When we had preliminary discussion of this with Maxim we both were inclined > to post it here and let it hang for a while to let dev list folks discuss > this idea in more details for just in case if we missed some usability > implications. > > Though now that you mentioned it I figured that proposed change is low risk > and easy to rollback, meaning we can do it the other way round: just merge > it now and keep in mind an option to revert if further discussion here > shows > that this way is wrong for some reason. > > If you prefer that we pick this way, changing priorities for TC inspections > could even be done as a part of another ticket involving this config file, > IGNITE-10422 - you can probably discuss with Maxim if he would be > comfortable with that (last time I checked he planned to do implementation > there). > > regards, Oleg > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Case sensitive indexes question.
Igniters, i found that ignite allow to create multiple case sensitive indexes with equal fields collection, i.e. no exception and warn here: CREATE INDEX \"title_idx\" ON books (title); CREATE INDEX \"tiTLE_IDX\" ON books (title); 1. in this case will be created two different index structures. 2. documentation [1] not clarify that quotation usage will create different ones and quotation absence will create index name in upper registry. 3. what index, query planner would be use? 4. and main question: why do we need this functional? i found that other vendors not allow such indexes [2] if we can`t change this in 2.x version due to some backward compatibility reasons, plz show them, i have no clue why we can`t change it in near release versions. [1] https://apacheignite-sql.readme.io/docs/create-index [2] https://oracle-base.com/articles/12c/multiple-indexes-on-same-set-of-columns-12cr1 thanks!
Re: Change code style inspections to use red mark for those used at Teamcity build checks (IGNITE-10450)
Hi Dmitry, When we had preliminary discussion of this with Maxim we both were inclined to post it here and let it hang for a while to let dev list folks discuss this idea in more details for just in case if we missed some usability implications. Though now that you mentioned it I figured that proposed change is low risk and easy to rollback, meaning we can do it the other way round: just merge it now and keep in mind an option to revert if further discussion here shows that this way is wrong for some reason. If you prefer that we pick this way, changing priorities for TC inspections could even be done as a part of another ticket involving this config file, IGNITE-10422 - you can probably discuss with Maxim if he would be comfortable with that (last time I checked he planned to do implementation there). regards, Oleg -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Historical rebalance
Guys, Another one idea. We can introduce additional update counter which is incremented by MVCC transactions right after executing operation (like is done for classic transactions). And we can use that counter for searching needed WAL records. Can it did the trick? P.S. Mentally I am trying to separate facilities providing transactions and durability. And it seems to me that those facilities are in different dimensions. ср, 28 нояб. 2018 г. в 16:26, Павлухин Иван : > > Sorry, if it was stated that a SINGLE transaction updates are applied > in a same order on all replicas then I have no questions so far. I > thought about reordering updates coming from different transactions. > > I have not got why we can assume that reordering is not possible. What > have I missed? > ср, 28 нояб. 2018 г. в 13:26, Павлухин Иван : > > > > Hi, > > > > Regarding Vladimir's new idea. > > > We assume that transaction can be represented as a set of independent > > > operations, which are applied in the same order on both primary and > > > backup nodes. > > I have not got why we can assume that reordering is not possible. What > > have I missed? > > вт, 27 нояб. 2018 г. в 14:42, Seliverstov Igor : > > > > > > Vladimir, > > > > > > I think I got your point, > > > > > > It should work if we do the next: > > > introduce two structures: active list (txs) and candidate list (updCntr -> > > > txn pairs) > > > > > > Track active txs, mapping them to actual update counter at update time. > > > On each next update put update counter, associated with previous update, > > > into a candidates list possibly overwrite existing value (checking txn) > > > On tx finish remove tx from active list only if appropriate update counter > > > (associated with finished tx) is applied. > > > On update counter update set the minimal update counter from the > > > candidates > > > list as a back-counter, clear the candidate list and remove an associated > > > tx from the active list if present. > > > Use back-counter instead of actual update counter in demand message. > > > > > > вт, 27 нояб. 2018 г. в 12:56, Seliverstov Igor : > > > > > > > Ivan, > > > > > > > > 1) The list is saved on each checkpoint, wholly (all transactions in > > > > active state at checkpoint begin). > > > > We need whole the list to get oldest transaction because after > > > > the previous oldest tx finishes, we need to get the following one. > > > > > > > > 2) I guess there is a description of how persistent storage works and > > > > how > > > > it restores [1] > > > > > > > > Vladimir, > > > > > > > > the whole list of what we going to store on checkpoint (updated): > > > > 1) Partition counter low watermark (LWM) > > > > 2) WAL pointer of earliest active transaction write to partition at the > > > > time the checkpoint have started > > > > 3) List of prepared txs with acquired partition counters (which were > > > > acquired but not applied yet) > > > > > > > > This way we don't need any additional info in demand message. Start > > > > point > > > > can be easily determined using stored WAL "back-pointer". > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-LocalRecoveryProcess > > > > > > > > > > > > вт, 27 нояб. 2018 г. в 11:19, Vladimir Ozerov : > > > > > > > >> Igor, > > > >> > > > >> Could you please elaborate - what is the whole set of information we > > > >> are > > > >> going to save at checkpoint time? From what I understand this should > > > >> be: > > > >> 1) List of active transactions with WAL pointers of their first writes > > > >> 2) List of prepared transactions with their update counters > > > >> 3) Partition counter low watermark (LWM) - the smallest partition > > > >> counter > > > >> before which there are no prepared transactions. > > > >> > > > >> And the we send to supplier node a message: "Give me all updates > > > >> starting > > > >> from that LWM plus data for that transactions which were active when I > > > >> failed". > > > >> > > > >> Am I right? > > > >> > > > >> On Fri, Nov 23, 2018 at 11:22 AM Seliverstov Igor > > > >> > > > >> wrote: > > > >> > > > >> > Hi Igniters, > > > >> > > > > >> > Currently I’m working on possible approaches how to implement > > > >> > historical > > > >> > rebalance (delta rebalance using WAL iterator) over MVCC caches. > > > >> > > > > >> > The main difficulty is that MVCC writes changes on tx active phase > > > >> > while > > > >> > partition update version, aka update counter, is being applied on tx > > > >> > finish. This means we cannot start iteration over WAL right from the > > > >> > pointer where the update counter updated, but should include updates, > > > >> which > > > >> > the transaction that updated the counter did. > > > >> > > > > >> > These updates may be much earlier than the point where the update > > > >> counter > > > >> > was updated, so we have to be able to identify the point where
Re: lost partition recovery with native persistence
Hi Roman, Also you could check if your problem is mentioned in another discussion related to lost partitions [1]. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Partition-Loss-Policies-issues-td37304.html ср, 28 нояб. 2018 г. в 19:31, novicr : > > Resending this to bubble up to the top of inbox. Would be good to hear > opinions on suggested functionality change. > > Thanks, > Roman > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ -- Best regards, Ivan Pavlukhin
Re: Change code style inspections to use red mark for those used at Teamcity build checks (IGNITE-10450)
Hi Oleg, Let's merge the change if we already have some, WDYT? Sincerely, Dmitriy Pavlov ср, 28 нояб. 2018 г. в 19:40, oignatenko : > Hello, > > When discussing issues involved in IGNITE-10399 we have got an idea to > change color highlighting of some code style inspections to make it easier > to see which can lead to problems at Teamcity. > > - Screen shot of how it is supposed to work: > > > https://issues.apache.org/jira/secure/attachment/12949868/IDEA.inspections.TC-bot.png > ^^^ Violations of inspections used by TC are shown as red in IDEA Error > Stripe while the rest remains yellow, looks easy to tell one from another. > (It's probably not very important but for the sake of completeness a thing > I > noticed when testing this is that having red inspections didn't block > compilation and execution of the code.) > > - JIRA ticket to change inspections: > https://issues.apache.org/jira/browse/IGNITE-10450 > > - Prior discussions related to inspections checks at Teamcity: > > > http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-tp27709.html > > Your feedback is welcome. > > TIA, > regards, Oleg > > PS. Special thanks to Maxim Muzafarov for help in brainstorming this matter > and in drafting JIRA ticket. > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Change code style inspections to use red mark for those used at Teamcity build checks (IGNITE-10450)
Hello, When discussing issues involved in IGNITE-10399 we have got an idea to change color highlighting of some code style inspections to make it easier to see which can lead to problems at Teamcity. - Screen shot of how it is supposed to work: https://issues.apache.org/jira/secure/attachment/12949868/IDEA.inspections.TC-bot.png ^^^ Violations of inspections used by TC are shown as red in IDEA Error Stripe while the rest remains yellow, looks easy to tell one from another. (It's probably not very important but for the sake of completeness a thing I noticed when testing this is that having red inspections didn't block compilation and execution of the code.) - JIRA ticket to change inspections: https://issues.apache.org/jira/browse/IGNITE-10450 - Prior discussions related to inspections checks at Teamcity: http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-tp27709.html Your feedback is welcome. TIA, regards, Oleg PS. Special thanks to Maxim Muzafarov for help in brainstorming this matter and in drafting JIRA ticket. -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: lost partition recovery with native persistence
Resending this to bubble up to the top of inbox. Would be good to hear opinions on suggested functionality change. Thanks, Roman -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[GitHub] ignite pull request #5528: Continuous query node restart
GitHub user novicr opened a pull request: https://github.com/apache/ignite/pull/5528 Continuous query node restart Add a test showing there is a problem with setting remote filter factory on continuous query. Steps to reproduce: 1. Start 4 node cluster 2. Create a ContinuousQuery 3. Set remote filter factory on the query (both factory and filter are Serializable) 4. Stop one server node 5. Start the node stopped in previous step In step 5 when starting the node `[2018-11-28 11:14:55,061][ERROR][tcp-disco-msg-worker-#40%continuous.CacheContinuousQueryRestartTest2%][TcpDiscoverySpi] Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-msg-worker-#40%continuous.CacheContinuousQueryRestartTest2%] java.lang.AssertionError at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandlerV2.getEventFilter(CacheContinuousQueryHandlerV2.java:104) ` The actual failing code: `assert rmtFilterFactory != null;` Looks like the filter factory is not propagated to the remote node. _Note:_ When I use setRemoteFilter() (which is now decommissioned) everything works as expected. You can merge this pull request into a Git repository by running: $ git pull https://github.com/novicr/ignite continuous-query-node-restart Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5528 commit 22d70239357a6e567c6d779588db56f44a0604b2 Author: novicr <43680008+novicr@...> Date: 2018-10-02T18:51:14Z Merge pull request #2 from apache/master pull latest into fork commit 99030d8b430b1cb94d41f5d2f7bd405505cd659b Author: novicr <43680008+novicr@...> Date: 2018-10-03T19:54:12Z add test to show sql query missing data when partitions are lost (#3) * add test to show sql query missing data when partitions are lost * add READ_WRITE_NONE lost partition policy * set partition loss policy * Revert: add READ_WRITE_NONE lost partition policy commit 71d79c553301c27811777eed55e4314c2f771af4 Author: novicr <43680008+novicr@...> Date: 2018-10-08T12:25:27Z Merge pull request #4 from apache/master update fork commit 85436863ce42b5032dcd19c76a472e7dd44f08fb Author: romannovichenok Date: 2018-11-28T16:04:18Z Add test to show remote filter factory missing when restarting node during continuous query. ---
[jira] [Created] (IGNITE-10450) In Ignite code style inspections increase level for those used at Teamcity build checks
Oleg Ignatenko created IGNITE-10450:
---
Summary: In Ignite code style inspections increase level for those
used at Teamcity build checks
Key: IGNITE-10450
URL: https://issues.apache.org/jira/browse/IGNITE-10450
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.6
Reporter: Oleg Ignatenko
Attachments: IDEA.inspections.TC-bot.png
Some of [Ignite code
style|https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines]
inspections are verified at Teamcity per IGNITE-9983. (Currently TC inspections
are "SizeReplaceableByIsEmpty", "UnusedImport", "MissingOverrideAnnotation",
"MissortedModifiers", "RedundantSuppression".)
Per discussion of issue IGNITE-10399 it looks like there is a room for
improvement here. Specifically, the problem occurred because it was too
difficult to find a new deviation that made TC inspections check fail because
it was buried among multiple similar looking but non-critical deviations in a
particular piece of old code
([PageMemoryImpl.java|https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/PageMemoryImpl.java]).
It would be more convenient if programmer could easier see subset of checks
that are used at Teamcity because this would allow to fix these earlier and
avoid cumbersome TC runs and failure reports analysis.
Technically this could be achieved by editing inspections config file and
increasing respective inspections level to {{ERROR}}. I briefly checked how it
would work in a "sandbox" project on my machine and it looked quite convenient:
violations of inspections used by TC were shown as red in Error Stripe while
the rest remained yellow, easy to see. (It's probably not very important but
for the sake of completeness a thing I noticed when testing is that having red
inspections didn't block compilation and execution of the code.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files
Eduard, Thank you very much for the discussion! Your algorithm looks much better for me too and easier to implement. I'll update appropriate process points on IEP page of the proposed rebalance procedure. On Tue, 27 Nov 2018 at 18:52, Eduard Shangareev wrote: > > So, after some discussion, I could describe another approach on how to > build consistent partition on the fly. > > 1. We make a checkpoint, fix the size of the partition in OffheapManager. > 2. After checkpoint finish, we start sending partition file (without any > lock) to the receiver from 0 to fixed size. > 3. Next checkpoints if they detect that they would override some pages of > transferring file should write the previous state of a page to a dedicated > file. > So, we would have a list of pages written 1 by 1, page id is written in the > page itself so we could determine page index. Let's name it log. > 4. When transfer finished checkpointer would stop updating log-file. Now we > are ready to send it to the receiver. > 5. On receiver side we start merging the dirty partition file with log > (updating it with pages from log-file). > > So, an advantage of this method: > - checkpoint-thread work couldn't increase more than twice; > - checkpoint-threads shouldn't wait for anything; > - in best case, we receive partition without any extra effort. > > > On Mon, Nov 26, 2018 at 8:54 PM Eduard Shangareev < > eduard.shangar...@gmail.com> wrote: > > > Maxim, > > > > I have looked through your algorithm of reading partition consistently. > > And I have some questions/comments. > > > > 1. The algorithm requires heavy synchronization between checkpoint-thread > > and new-approach-rebalance-threads, > > because you need strong guarantees to not start writing or reading to > > chunk which was updated or started reading by the counterpart. > > > > 2. Also, if we have started transferring this chunk in original partition > > couldn't be updated by checkpoint-threads. They should wait for transfer > > finishing. > > > > 3. If sending is slow and partition is updated then in worst case > > checkpoint-threads would create the whole copy of the partition. > > > > So, what we have: > > -on every page write checkpoint-thread should synchronize with > > new-approach-rebalance-threads; > > -checkpoint-thread should do extra-work, sometimes this could be as huge > > as copying the whole partition. > > > > > > On Fri, Nov 23, 2018 at 2:55 PM Ilya Kasnacheev > > wrote: > > > >> Hello! > >> > >> This proposal will also happily break my compression-with-dictionary patch > >> since it relies currently on only having local dictionaries. > >> > >> However, when you have compressed data, maybe speed boost is even greater > >> with your approach. > >> > >> Regards, > >> -- > >> Ilya Kasnacheev > >> > >> > >> пт, 23 нояб. 2018 г. в 13:08, Maxim Muzafarov : > >> > >> > Igniters, > >> > > >> > > >> > I'd like to take the next step of increasing the Apache Ignite with > >> > enabled persistence rebalance speed. Currently, the rebalancing > >> > procedure doesn't utilize the network and storage device throughout to > >> > its full extent even with enough meaningful values of > >> > rebalanceThreadPoolSize property. As part of the previous discussion > >> > `How to make rebalance faster` [1] and IEP-16 [2] Ilya proposed an > >> > idea [3] of transferring cache partition files over the network. > >> > From my point, the case to which this type of rebalancing procedure > >> > can bring the most benefit – is adding a completely new node or set of > >> > new nodes to the cluster. Such a scenario implies fully relocation of > >> > cache partition files to the new node. To roughly estimate the > >> > superiority of partition file transmitting over the network the native > >> > Linux scp\rsync commands can be used. My test environment showed the > >> > result of the new approach as 270 MB/s vs the current 40 MB/s > >> > single-threaded rebalance speed. > >> > > >> > > >> > I've prepared the design document IEP-28 [4] and accumulated all the > >> > process details of a new rebalance approach on that page. Below you > >> > can find the most significant details of the new rebalance procedure > >> > and components of the Apache Ignite which are proposed to change. > >> > > >> > Any feedback is very appreciated. > >> > > >> > > >> > *PROCESS OVERVIEW* > >> > > >> > The whole process is described in terms of rebalancing single cache > >> > group and partition files would be rebalanced one-by-one: > >> > > >> > 1. The demander node sends the GridDhtPartitionDemandMessage to the > >> > supplier node; > >> > 2. When the supplier node receives GridDhtPartitionDemandMessage and > >> > starts the new checkpoint process; > >> > 3. The supplier node creates empty the temporary cache partition file > >> > with .tmp postfix in the same cache persistence directory; > >> > 4. The supplier node splits the whole cache partition file into > >> > virtual chunks of predefined size (multiply to the
Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files
Vladimir, > Another thing to consider is MVCC - under load partitions may have a lot of > entries which are not visible to anyone and is about to be removed, so there > is no need to transfer them at all. Please share more details about how these entries occur. When they become invisible? From what I've found, they appear after entry deletion, right? It looks that when checkpoint ends these entries will be gone, but I can mistake here. I'll try to take it into account and will update the IEP page. > Do we have any analysis on where time is really spent during current > rebalance implementation? Yes, I've collected some statistics about the rebalancing procedure and also I've tested it with different types of available rebalance properties. The wiki page [1] of current rebalancing procedure limitations and advantages of the new one was created by me. I have not published yet everything measurements that I have, but, please, look at the graph placed on that page. We have higher CPU consumption on the demander node rather than on the supplier node. This is all without any additional load. I think it shows us that saving entries one by one is not the right strategy for the cache data balancing. Therefore, I think we have some options here (you already mentioned some of them): a batch entries processing, optimization internal data structures, or avoid it at all by transferring stored files. We already have tickets for the fuzzy free list implementation [2] and the batch entries processing [3]. At that time in the past and now these changes looks to me more complex and risky (maybe I'm missing something and they are easier). I think it's acceptable to start (see the next comment - why) the cluster rebalancing procedure optimization from persistence enabled perspective by prototyping proposed approach. > But it is rather complex, doesn't help in-memory mode, and will not work for > other storage formats which we may potentially have in future (e.g. > tablespace approach). You are not actually right here. Yes, this proposal is only for clusters with enabled persistence, but don't consider these changes as a huge monolithic update. From my point, It's a set of independent features that will give Apache Ignite a competitive advantage. For instance, changes in Chekpointer will give us an opportunity to save (over the network or a direct copy to some file) data snapshots of persisted files under checkpoint at some point in time. Or another example, changes in CommunicationSpi will allow us to have a channel connection between any pair of nodes for any needs (e.g. copying any files using zero-copy algorithm without node CPU resources consumption or any binary data as well). I've read your topic about remaining cache groups to the tablespace and I very like this idea. I can say that the new type of storage organization "file-segment-extent" will lead us to change only Preloader implementation (or write another one, for each type of storage organization), other parts of current proposal will work right out of the box. I think we can get a huge rebalance speed improvement on very fast SSDs even more than with batched data processing on the demander side or fuzzy free list implementation. I'll try to prototype the current solution and recheck all measurements. Please correct me where I am wrong. [1] https://cwiki.apache.org/confluence/display/IGNITE/Rebalance+peer-2-peer [2] https://issues.apache.org/jira/browse/IGNITE-9520 [3] https://issues.apache.org/jira/browse/IGNITE-7935 On Fri, 23 Nov 2018 at 14:03, Vladimir Ozerov wrote: > > Maxim, > > Do we have any analysis on where time is really spent during current > rebalance implementation? Proposed change may work very well. But it is > rather complex, doesn't help in-memory mode, and will not work for other > storage formats which we may potentially have in future (e.g. tablespace > approach). Another thing to consider is MVCC - under load partitions may > have a lot of entries which are not visible to anyone and is about to be > removed, so there is no need to transfer them at all. > > Did we investigate any universal and less intrusive approaches to rebalance > speedup before that? For example: > - batched data block reads on supplier > - iteration over partition rather than cache data tree on supplier > - batched data save on demander > - delayed free list and index re-build in demander > > Vladimir. > > On Fri, Nov 23, 2018 at 1:08 PM Maxim Muzafarov wrote: > > > Igniters, > > > > > > I'd like to take the next step of increasing the Apache Ignite with > > enabled persistence rebalance speed. Currently, the rebalancing > > procedure doesn't utilize the network and storage device throughout to > > its full extent even with enough meaningful values of > > rebalanceThreadPoolSize property. As part of the previous discussion > > `How to make rebalance faster` [1] and IEP-16 [2] Ilya proposed an > > idea [3] of transferring cache partition files over
[GitHub] ignite pull request #5526: IGNITE-10449: Fix javadoc and typos.
GitHub user dmitrievanthony opened a pull request: https://github.com/apache/ignite/pull/5526 IGNITE-10449: Fix javadoc and typos. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10449 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5526 commit 42d44e930ecd178da7d29f901fbc879fed4437db Author: dmitrievanthony Date: 2018-11-28T15:30:18Z IGNITE-10449: Fix javadoc and typos. ---
[jira] [Created] (IGNITE-10449) ML: Fix inspection issues
Anton Dmitriev created IGNITE-10449: --- Summary: ML: Fix inspection issues Key: IGNITE-10449 URL: https://issues.apache.org/jira/browse/IGNITE-10449 Project: Ignite Issue Type: Improvement Components: ml Affects Versions: 2.8 Reporter: Anton Dmitriev Assignee: Anton Dmitriev Fix For: 2.8 Static inspections found several issues: missed javadoc, typos, etc. The goal of this task is to fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)
Denis I would wait for running queries view first. ср, 28 нояб. 2018 г. в 1:57, Denis Magda : > Vladimir, > > Please see inline > > On Mon, Nov 19, 2018 at 8:23 AM Vladimir Ozerov > wrote: > > > Denis, > > > > I partially agree with you. But there are several problem with syntax > > proposed by you: > > 1) This is harder to implement technically - more parsing logic to > > implement. Ok, this is our internal problem, users do not care about it > > 2) User will have to consult to docs in any case > > > > Two of these are not a big deal. We just need to invest more time for > development and during the design phase so that people need to consult the > docs rarely. > > > > 3) "nodeId" is not really node ID. For Ignite users node ID is UUID. In > our > > case this is node order, and we intentionally avoided any naming here. > > > > Let's use a more loose name such as "node". > > > > Query is just identified by a string, no more than that > > 4) Proposed syntax is more verbose and open ways for misuse. E.g. what is > > "KILL QUERY WHERE queryId=1234"? > > > > I am not 100% satisfied with both variants, but the first one looks > simpler > > to me. Remember, that user will not guess query ID. Instead, he will get > > the list of running queries with some other syntax. What we need to > > understand for now is how this syntax will look like. I think that we > > should implement getting list of running queries, and only then start > > working on cancellation. > > > > That's a good point. Syntax of both running and killing queires commands > should be tightly coupled. We're going to name a column if running queries > IDs somehow anyway and that name might be resued in the WHERE clause of > KILL. > > Should we discuss the syntax in a separate thread? > > -- > Denis > > > > > Vladimir. > > > > > > On Mon, Nov 19, 2018 at 7:02 PM Denis Mekhanikov > > wrote: > > > > > Guys, > > > > > > Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me. > > > I'm going to look up in documentation, which parameter goes first in > this > > > query every time I use it. > > > I like the syntax, that Igor suggested more. > > > Will it be better if we make *nodeId* and *queryId *named properties? > > > > > > Something like this: > > > KILL QUERY WHERE nodeId=25 and queryId=1234 > > > > > > Denis > > > > > > пт, 16 нояб. 2018 г. в 14:12, Юрий : > > > > > > > I fully agree with last sentences and can start to implement this > part. > > > > > > > > Guys, thanks for your productive participate at discussion. > > > > > > > > пт, 16 нояб. 2018 г. в 2:53, Denis Magda : > > > > > > > > > Vladimir, > > > > > > > > > > Thanks, make perfect sense to me. > > > > > > > > > > > > > > > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov < > > voze...@gridgain.com > > > > > > > > > wrote: > > > > > > > > > > > Denis, > > > > > > > > > > > > The idea is that QueryDetailMetrics will be exposed through > > separate > > > > > > "historical" SQL view in addition to current API. So we are on > the > > > same > > > > > > page here. > > > > > > > > > > > > As far as query ID I do not see any easy way to operate on a > single > > > > > integer > > > > > > value (even 64bit). This is distributed system - we do not want > to > > > have > > > > > > coordination between nodes to get query ID. And coordination is > the > > > > only > > > > > > possible way to get sexy "long". Instead, I would propose to form > > ID > > > > from > > > > > > node order and query counter within node. This will be (int, > long) > > > > pair. > > > > > > For use convenience we may convert it to a single string, e.g. > > > > > > "[node_order].[query_counter]". Then the syntax would be: > > > > > > > > > > > > KILL QUERY '25.1234'; // Kill query 1234 on node 25 > > > > > > KILL QUERY '25.*; // Kill all queries on the node 25 > > > > > > > > > > > > Makes sense? > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > On Wed, Nov 14, 2018 at 1:25 PM Denis Magda > > > wrote: > > > > > > > > > > > > > Yury, > > > > > > > > > > > > > > As I understand you mean that the view should contains both > > running > > > > and > > > > > > > > finished queries. If be honest for the view I was going to > use > > > just > > > > > > > queries > > > > > > > > running right now. For finished queries I thought about > another > > > > view > > > > > > with > > > > > > > > another set of fields which should include I/O related ones. > Is > > > it > > > > > > works? > > > > > > > > > > > > > > > > > > > > > Got you, so if only running queries are there then your initial > > > > > proposal > > > > > > > makes total sense. Not sure we need a view of the finished > > queries. > > > > It > > > > > > will > > > > > > > be possible to analyze them through the updated DetailedMetrics > > > > > approach, > > > > > > > won't it? > > > > > > > > > > > > > > For "KILL QUERY node_id query_id" node_id required as part of > > > unique > > > > > key > > > > > > > > of query and help understand Ignite which node
[jira] [Created] (IGNITE-10448) MVCC: NPE on data page eviction.
Roman Kondakov created IGNITE-10448:
---
Summary: MVCC: NPE on data page eviction.
Key: IGNITE-10448
URL: https://issues.apache.org/jira/browse/IGNITE-10448
Project: Ignite
Issue Type: Bug
Components: mvcc
Reporter: Roman Kondakov
NPE occurred during page eviction process.
Reproducer: {{PageEvictionMultinodeMixedRegionsTest}}.
StackTrace:
{noformat}
javax.cache.CacheException: class
org.apache.ignite.transactions.TransactionRollbackException: Transaction has
been rolled back: 83e7f9a5761--093b-7d30--0005
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1108)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
at
org.apache.ignite.internal.processors.cache.eviction.paged.PageEvictionMultinodeAbstractTest.createCacheAndTestEvcition(PageEvictionMultinodeAbstractTest.java:101)
at
org.apache.ignite.internal.processors.cache.eviction.paged.PageEvictionMultinodeAbstractTest.testPageEviction(PageEvictionMultinodeAbstractTest.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at
org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:150)
at
org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2104)
at
org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2119)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.transactions.TransactionRollbackException:
Transaction has been rolled back:
83e7f9a5761--093b-7d30--0005
at
org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:923)
at
org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:921)
... 15 more
Caused by: class
org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException:
Transaction has been rolled back:
83e7f9a5761--093b-7d30--0005
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4299)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2520)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2501)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2478)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
... 12 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to update
backup node: [localNodeId=0d817370-17fe-46f1-85ef-ea74b6f1,
remoteNodeId=ebab34f3-abbc-47e9-90fa-a48a8260]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:1012)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2340)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1200(GridDhtTransactionalCacheAdapter.java:112)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:257)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:255)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1059)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:584)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:383)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:309)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:100)
at
[GitHub] ignite pull request #5525: IGNITE-10291
GitHub user devozerov opened a pull request: https://github.com/apache/ignite/pull/5525 IGNITE-10291 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10291-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5525 commit 10f5f9e328c081953462c498b58edad636e307fb Author: devozerov Date: 2018-11-28T09:28:40Z WIP. commit b35634bf58a3deb903f4ba9590c36eaa16d8fa4a Author: devozerov Date: 2018-11-28T09:51:32Z Minors. commit 898e82a27dd8a0ec7773eaa5b1e641c12c6c64cd Author: devozerov Date: 2018-11-28T12:27:55Z Implemented. commit 48601580e0a87bf2c5c759f736fce3543a5e105c Author: devozerov Date: 2018-11-28T12:29:37Z Minors. commit 8a2a8610a8439b8fedc3271153379d0d57058128 Author: devozerov Date: 2018-11-28T13:55:52Z It works. commit 7749dc7e34ffadfeee43d79e4098fdbe8085ddeb Author: devozerov Date: 2018-11-28T14:00:15Z Removed "exists". commit 24092fc6828e9246873814431dddfeb142fa1c8c Author: devozerov Date: 2018-11-28T14:04:01Z Minors. commit d0c2ee7a01bb202315d7c6ffc917b9ffb0d0bfc8 Author: devozerov Date: 2018-11-28T14:07:24Z Minors. ---
[jira] [Created] (IGNITE-10447) thin nodejs: can't execute example AuthTlsExample.js
Stepan Pilschikov created IGNITE-10447:
--
Summary: thin nodejs: can't execute example AuthTlsExample.js
Key: IGNITE-10447
URL: https://issues.apache.org/jira/browse/IGNITE-10447
Project: Ignite
Issue Type: Bug
Components: thin client
Affects Versions: 2.7
Reporter: Stepan Pilschikov
Trying to run script from nodejs/examples/AuthTlsExample.js but connection
failed
Output:
{code}
$ ignite/modules/platforms/nodejs/examples$ node AuthTlsExample.js
Client is stopped
[localhost:10800] Connection failed: Error: Client network socket disconnected
before secure TLS connection was established
ERROR: [localhost:10800] Connection failed: Error: Client network socket
disconnected before secure TLS connection was established
{code}
config.xml
{code}
http://www.springframework.org/schema/beans;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Thin clients all in one
Hello again
If NodeJS sources found that example AuthTlsExample.js throwing exception
during execution
Output and grid configuration in
https://issues.apache.org/jira/browse/IGNITE-10447
Can someone have a look at it?
вс, 25 нояб. 2018 г. в 19:11, Stepan Pilschikov :
> My bad,
> You right
>
> вс, 25 нояб. 2018 г. в 05:37, Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com>:
>
>> Stepan,
>>
>> AFAIK Map type did always behave correctly on client side, as it does
>> now. This is a corresponding piece of my test suite:
>>
>> ```
>> def test_put_get_map(client):
>>
>> cache = client.get_or_create_cache('test_map_cache')
>>
>> cache.put(
>> 'test_map',
>> (
>> MapObject.HASH_MAP,
>> {
>> (123, IntObject): 'test_data',
>> 456: ((1, [456, 'inner_test_string', 789]),
>> CollectionObject),
>> 'test_key': 32.4,
>> }
>> ),
>> value_hint=MapObject
>> )
>> value = cache.get('test_map')
>> assert value == (MapObject.HASH_MAP, {
>> 123: 'test_data',
>> 456: (1, [456, 'inner_test_string', 789]),
>> 'test_key': 32.4,
>> })
>>
>> ```
>>
>> Or is there another, more specific problem with maps?
>>
>> Dmitry
>>
>> On 11/25/18 3:56 AM, Stepan Pilschikov wrote:
>> > Dmitry,
>> >
>> > Great, checked, now all things woks well
>> > Hope that Igor made review for this PR
>> >
>> > But what about Maps? Looks like different ticket? or it can be done in
>> same
>> > ticket scope?
>> >
>> > пт, 23 нояб. 2018 г. в 23:58, Dmitry Melnichuk <
>> > dmitry.melnic...@nobitlost.com>:
>> >
>> >> Stepan,
>> >>
>> >> Sorry, I forgot to update from upstream prior to start working on this
>> >> issue, and thus brought a regression. My bad. Just merged with the
>> >> latest master. Please, check it out again.
>> >>
>> >> Dmitry
>> >>
>> >> On 11/24/18 1:37 AM, Stepan Pilschikov wrote:
>> >>> Dmitry,
>> >>>
>> >>> Iv checked and its actually work
>> >>> But a specially in this branch i found another bug
>> >>> Please look at my last comment:
>> >>>
>> >>
>> https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285
>> >>>
>> >>> пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk <
>> >>> dmitry.melnic...@nobitlost.com>:
>> >>>
>> Stepan,
>>
>> Thank you for your great job in evaluating Python thin client, as
>> well
>> as other thin clients.
>>
>> There was indeed a bug in Python client regarding the handling of
>> type
>> hints in Collection type. I created a fix and did a PR under
>> IGNITE-10358 task, but the same PR is also fixes the problem in
>> IGNITE-10230 task.
>>
>> As of handling the type mapping in gists you provided, I left
>> comments
>> on both tasks.
>>
>> Dmitry
>>
>> On 11/21/18 6:37 PM, Stepan Pilschikov wrote:
>> > Dmitry, Alexey
>> >
>> > Thank you for help, this answers help me a lot with understanding
>> how
>> > clients are work
>> >
>> > Not so long time ago i met problem which is have expected behavior,
>> but
>> its
>> > may broke some workflows in future for some users
>> >
>> > Its all about not specified data types in collections and map's
>> > All description and examples in
>> > https://issues.apache.org/jira/browse/IGNITE-10358
>> >
>> > Dmitry, can you have a quick look at it and maybe in future somehow
>> fix
>> it?
>> >
>> > пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
>> > dmitry.melnic...@nobitlost.com>:
>> >
>> >> Stepan!
>> >>
>> >> TL/DR: what you got with Python client in your gist is an intended
>> >> behavior.
>> >>
>> >> Explanation: As per docs, Object array contains of type ID (which
>> is
>> >> defaults to -1) and an array of objects.
>> >>
>> >>
>> >>
>>
>> >>
>> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array
>> >>
>> >> Your gist might be fixed accordingly:
>> >>
>> >> ```
>> >> from pyignite import Client
>> >> from pyignite.datatypes import *
>> >>
>> >> OBJECT_ARRAY_TYPE_ID = -1
>> >> OBJECT_ARRAY_CONTENTS = [1, 2]
>> >>
>> >> client = Client()
>> >> client.connect('127.0.0.1', 10800)
>> >> cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
>> >> cache.put(
>> >> 1,
>> >> (OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
>> >> key_hint=IntObject,
>> >> value_hint=ObjectArrayObject,
>> >> )
>> >>
>> >> # Python output: print(cache.get(1))
>> >> # (-1, [1, 2])
>> >> ```
>> >>
>> >> The situation is similar with Map and Collection, they have types.
>> >> Types
>> >> and type IDs are mostly useless in Python, but I left them for
>> >>
[jira] [Created] (IGNITE-10446) control.sh --cache idle_verify fail with NPE when node left grid
ARomantsov created IGNITE-10446: --- Summary: control.sh --cache idle_verify fail with NPE when node left grid Key: IGNITE-10446 URL: https://issues.apache.org/jira/browse/IGNITE-10446 Project: Ignite Issue Type: Bug Affects Versions: 2.7 Reporter: ARomantsov Fix For: 2.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10445) Move Tx state to offheap.
Andrew Mashenkov created IGNITE-10445: - Summary: Move Tx state to offheap. Key: IGNITE-10445 URL: https://issues.apache.org/jira/browse/IGNITE-10445 Project: Ignite Issue Type: Improvement Components: cache Reporter: Andrew Mashenkov For now TxManager track tx state in on-heap bounded map structure. Map size can be changed with system prop-ty IGNITE_MAX_COMPLETED_TX_COUNT. Let's move tx structures to offheap like it is done for Mvcc transaction (TxLog). Also seems, we can use tx writeVersion as a commitVersion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Historical rebalance
Sorry, if it was stated that a SINGLE transaction updates are applied in a same order on all replicas then I have no questions so far. I thought about reordering updates coming from different transactions. > I have not got why we can assume that reordering is not possible. What have I missed? ср, 28 нояб. 2018 г. в 13:26, Павлухин Иван : > > Hi, > > Regarding Vladimir's new idea. > > We assume that transaction can be represented as a set of independent > > operations, which are applied in the same order on both primary and backup > > nodes. > I have not got why we can assume that reordering is not possible. What > have I missed? > вт, 27 нояб. 2018 г. в 14:42, Seliverstov Igor : > > > > Vladimir, > > > > I think I got your point, > > > > It should work if we do the next: > > introduce two structures: active list (txs) and candidate list (updCntr -> > > txn pairs) > > > > Track active txs, mapping them to actual update counter at update time. > > On each next update put update counter, associated with previous update, > > into a candidates list possibly overwrite existing value (checking txn) > > On tx finish remove tx from active list only if appropriate update counter > > (associated with finished tx) is applied. > > On update counter update set the minimal update counter from the candidates > > list as a back-counter, clear the candidate list and remove an associated > > tx from the active list if present. > > Use back-counter instead of actual update counter in demand message. > > > > вт, 27 нояб. 2018 г. в 12:56, Seliverstov Igor : > > > > > Ivan, > > > > > > 1) The list is saved on each checkpoint, wholly (all transactions in > > > active state at checkpoint begin). > > > We need whole the list to get oldest transaction because after > > > the previous oldest tx finishes, we need to get the following one. > > > > > > 2) I guess there is a description of how persistent storage works and how > > > it restores [1] > > > > > > Vladimir, > > > > > > the whole list of what we going to store on checkpoint (updated): > > > 1) Partition counter low watermark (LWM) > > > 2) WAL pointer of earliest active transaction write to partition at the > > > time the checkpoint have started > > > 3) List of prepared txs with acquired partition counters (which were > > > acquired but not applied yet) > > > > > > This way we don't need any additional info in demand message. Start point > > > can be easily determined using stored WAL "back-pointer". > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-LocalRecoveryProcess > > > > > > > > > вт, 27 нояб. 2018 г. в 11:19, Vladimir Ozerov : > > > > > >> Igor, > > >> > > >> Could you please elaborate - what is the whole set of information we are > > >> going to save at checkpoint time? From what I understand this should be: > > >> 1) List of active transactions with WAL pointers of their first writes > > >> 2) List of prepared transactions with their update counters > > >> 3) Partition counter low watermark (LWM) - the smallest partition counter > > >> before which there are no prepared transactions. > > >> > > >> And the we send to supplier node a message: "Give me all updates starting > > >> from that LWM plus data for that transactions which were active when I > > >> failed". > > >> > > >> Am I right? > > >> > > >> On Fri, Nov 23, 2018 at 11:22 AM Seliverstov Igor > > >> wrote: > > >> > > >> > Hi Igniters, > > >> > > > >> > Currently I’m working on possible approaches how to implement > > >> > historical > > >> > rebalance (delta rebalance using WAL iterator) over MVCC caches. > > >> > > > >> > The main difficulty is that MVCC writes changes on tx active phase > > >> > while > > >> > partition update version, aka update counter, is being applied on tx > > >> > finish. This means we cannot start iteration over WAL right from the > > >> > pointer where the update counter updated, but should include updates, > > >> which > > >> > the transaction that updated the counter did. > > >> > > > >> > These updates may be much earlier than the point where the update > > >> counter > > >> > was updated, so we have to be able to identify the point where the > > >> > first > > >> > update happened. > > >> > > > >> > The proposed approach includes: > > >> > > > >> > 1) preserve list of active txs, sorted by the time of their first > > >> > update > > >> > (using WAL ptr of first WAL record in tx) > > >> > > > >> > 2) persist this list on each checkpoint (together with TxLog for > > >> example) > > >> > > > >> > 4) send whole active tx list (transactions which were in active state > > >> > at > > >> > the time the node was crushed, empty list in case of graceful node > > >> stop) as > > >> > a part of partition demand message. > > >> > > > >> > 4) find a checkpoint where the earliest tx exists in persisted txs and > > >> use > > >> > saved WAL ptr as a start point or apply current approach
[jira] [Created] (IGNITE-10444) MVCC: CacheMvccTxFastFinishTest fails.
Andrew Mashenkov created IGNITE-10444: - Summary: MVCC: CacheMvccTxFastFinishTest fails. Key: IGNITE-10444 URL: https://issues.apache.org/jira/browse/IGNITE-10444 Project: Ignite Issue Type: Bug Components: mvcc Reporter: Andrew Mashenkov Seems, read-only transaction doesn't creates prepare\finish future as classic tx do. Let's check correct invariants in mvcc version of CacheTxFastFinishTest test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #5243: IGNITE-10080
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5243 ---
[GitHub] ignite pull request #5524: fix index upper case
GitHub user zstan opened a pull request: https://github.com/apache/ignite/pull/5524 fix index upper case You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-14511 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5524.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5524 commit d32138695f5309d9e457df996fe1bd6c2a447ea2 Author: Evgeny Stanilovskiy Date: 2018-11-28T12:46:19Z fix index upper case ---
[jira] [Created] (IGNITE-10443) Fix flaky GridCommandHandlerTest.testKillHangingLocalTransactions
Alexei Scherbakov created IGNITE-10443: -- Summary: Fix flaky GridCommandHandlerTest.testKillHangingLocalTransactions Key: IGNITE-10443 URL: https://issues.apache.org/jira/browse/IGNITE-10443 Project: Ignite Issue Type: Test Reporter: Alexei Scherbakov Assignee: Alexei Scherbakov Fix For: 2.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10442) Failed checks on successful tasks canceling
Ivan Fedotov created IGNITE-10442: - Summary: Failed checks on successful tasks canceling Key: IGNITE-10442 URL: https://issues.apache.org/jira/browse/IGNITE-10442 Project: Ignite Issue Type: Test Reporter: Ivan Fedotov Assignee: Ivan Fedotov [testFailNodesOnCanceledTask|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-1511425443970810161=testDetails_IgniteTests24Java8=%3Cdefault%3E] seems flaky. On of the reason - "Possible thread pool starvation detected". It is necessary to investigate test and fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10441) Fluent API refactoring.
Artem Malykh created IGNITE-10441:
-
Summary: Fluent API refactoring.
Key: IGNITE-10441
URL: https://issues.apache.org/jira/browse/IGNITE-10441
Project: Ignite
Issue Type: Improvement
Components: ml
Reporter: Artem Malykh
Assignee: Artem Malykh
In many classes we have fluent API ("with*" methods). We have following
problem: these methods should return exactly instance of it's own class
(otherwise we'll have problems with subclasses, more precisely, if with method
is declared in class A and we have class B extending A, with method (if we do
not override it) will return A). Currently we opted to override "with" methods
in subclasses. There is one solution which is probably more elegant, but
involves relatively complex generics construction which reduces readability:
{code:java}
class A> {
Self withX(X x) {
this.x = x;
return (Self)this;
}
class B> extends A {
// No need to override "withX" here
Self withY(Y y) {
this.y = y;
return(Self)this;
}
}
class C> extends B {
// No need to override "withX" and "withY" methods here.
}
//... etc
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-10440) Analyse test suites for possible acceleration
Alexey Platonov created IGNITE-10440: Summary: Analyse test suites for possible acceleration Key: IGNITE-10440 URL: https://issues.apache.org/jira/browse/IGNITE-10440 Project: Ignite Issue Type: Improvement Reporter: Alexey Platonov Assignee: Alexey Platonov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10439) [ML] Examples of DBSCAN
Yury Babak created IGNITE-10439: --- Summary: [ML] Examples of DBSCAN Key: IGNITE-10439 URL: https://issues.apache.org/jira/browse/IGNITE-10439 Project: Ignite Issue Type: Sub-task Reporter: Yury Babak We need an example for DBSCAN usage -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10438) [ML] DBSCAN
Yury Babak created IGNITE-10438: --- Summary: [ML] DBSCAN Key: IGNITE-10438 URL: https://issues.apache.org/jira/browse/IGNITE-10438 Project: Ignite Issue Type: New Feature Components: ml Reporter: Yury Babak Density-based spatial clustering of applications with noise (DBSCAN) [wiki description|https://en.wikipedia.org/wiki/DBSCAN] We could test this algorithm on TWO_CLASSED_IRIS and IRIS (see MLSandboxDatasets enum) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #5517: IGNITE-10298 Cover possible deadlock in case of c...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5517 ---
[GitHub] ignite pull request #5504: Ignite gc 371 2
Github user AlexDel closed the pull request at: https://github.com/apache/ignite/pull/5504 ---
Re: Historical rebalance
Hi, Regarding Vladimir's new idea. > We assume that transaction can be represented as a set of independent > operations, which are applied in the same order on both primary and backup > nodes. I have not got why we can assume that reordering is not possible. What have I missed? вт, 27 нояб. 2018 г. в 14:42, Seliverstov Igor : > > Vladimir, > > I think I got your point, > > It should work if we do the next: > introduce two structures: active list (txs) and candidate list (updCntr -> > txn pairs) > > Track active txs, mapping them to actual update counter at update time. > On each next update put update counter, associated with previous update, > into a candidates list possibly overwrite existing value (checking txn) > On tx finish remove tx from active list only if appropriate update counter > (associated with finished tx) is applied. > On update counter update set the minimal update counter from the candidates > list as a back-counter, clear the candidate list and remove an associated > tx from the active list if present. > Use back-counter instead of actual update counter in demand message. > > вт, 27 нояб. 2018 г. в 12:56, Seliverstov Igor : > > > Ivan, > > > > 1) The list is saved on each checkpoint, wholly (all transactions in > > active state at checkpoint begin). > > We need whole the list to get oldest transaction because after > > the previous oldest tx finishes, we need to get the following one. > > > > 2) I guess there is a description of how persistent storage works and how > > it restores [1] > > > > Vladimir, > > > > the whole list of what we going to store on checkpoint (updated): > > 1) Partition counter low watermark (LWM) > > 2) WAL pointer of earliest active transaction write to partition at the > > time the checkpoint have started > > 3) List of prepared txs with acquired partition counters (which were > > acquired but not applied yet) > > > > This way we don't need any additional info in demand message. Start point > > can be easily determined using stored WAL "back-pointer". > > > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-LocalRecoveryProcess > > > > > > вт, 27 нояб. 2018 г. в 11:19, Vladimir Ozerov : > > > >> Igor, > >> > >> Could you please elaborate - what is the whole set of information we are > >> going to save at checkpoint time? From what I understand this should be: > >> 1) List of active transactions with WAL pointers of their first writes > >> 2) List of prepared transactions with their update counters > >> 3) Partition counter low watermark (LWM) - the smallest partition counter > >> before which there are no prepared transactions. > >> > >> And the we send to supplier node a message: "Give me all updates starting > >> from that LWM plus data for that transactions which were active when I > >> failed". > >> > >> Am I right? > >> > >> On Fri, Nov 23, 2018 at 11:22 AM Seliverstov Igor > >> wrote: > >> > >> > Hi Igniters, > >> > > >> > Currently I’m working on possible approaches how to implement historical > >> > rebalance (delta rebalance using WAL iterator) over MVCC caches. > >> > > >> > The main difficulty is that MVCC writes changes on tx active phase while > >> > partition update version, aka update counter, is being applied on tx > >> > finish. This means we cannot start iteration over WAL right from the > >> > pointer where the update counter updated, but should include updates, > >> which > >> > the transaction that updated the counter did. > >> > > >> > These updates may be much earlier than the point where the update > >> counter > >> > was updated, so we have to be able to identify the point where the first > >> > update happened. > >> > > >> > The proposed approach includes: > >> > > >> > 1) preserve list of active txs, sorted by the time of their first update > >> > (using WAL ptr of first WAL record in tx) > >> > > >> > 2) persist this list on each checkpoint (together with TxLog for > >> example) > >> > > >> > 4) send whole active tx list (transactions which were in active state at > >> > the time the node was crushed, empty list in case of graceful node > >> stop) as > >> > a part of partition demand message. > >> > > >> > 4) find a checkpoint where the earliest tx exists in persisted txs and > >> use > >> > saved WAL ptr as a start point or apply current approach in case the > >> active > >> > tx list (sent on previous step) is empty > >> > > >> > 5) start iteration. > >> > > >> > Your thoughts? > >> > > >> > Regards, > >> > Igor > >> > > -- Best regards, Ivan Pavlukhin
[GitHub] ignite pull request #5523: 10437 test flakiness
GitHub user SomeFire opened a pull request: https://github.com/apache/ignite/pull/5523 10437 test flakiness You can merge this pull request into a Git repository by running: $ git pull https://github.com/SomeFire/ignite IGNITE-10437-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5523.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5523 commit 756aabf23be89c13e43abdcd5913fa707b8f3040 Author: Dmitrii Ryabov Date: 2018-11-28T09:27:08Z IGNITE-10437 GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThreadWithCoalescing is flaky commit 76b8c1d8c6dedf4c56f385a9912b90c4d62bb4d6 Author: Dmitrii Ryabov Date: 2018-11-28T09:31:21Z test flakiness ---
[GitHub] ignite pull request #5522: IGNITE-10437 GridCacheWriteBehindStoreMultithread...
GitHub user SomeFire opened a pull request: https://github.com/apache/ignite/pull/5522 IGNITE-10437 GridCacheWriteBehindStoreMultithreadedSelfTest.testFlush⦠â¦FromTheSameThreadWithCoalescing is flaky You can merge this pull request into a Git repository by running: $ git pull https://github.com/SomeFire/ignite IGNITE-10437 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5522.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5522 commit 756aabf23be89c13e43abdcd5913fa707b8f3040 Author: Dmitrii Ryabov Date: 2018-11-28T09:27:08Z IGNITE-10437 GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThreadWithCoalescing is flaky ---
Re: Apache Ignite 2.7. Last Mile
Fixed. Thank you for noting it. On Wed, Nov 28, 2018 at 6:22 AM Alexey Kuznetsov wrote: > Hi, > > We found a regression https://issues.apache.org/jira/browse/IGNITE-10432 > > Please take a look. > > -- > Alexey Kuznetsov >
[GitHub] ignite pull request #5521: IGNITE-10435
GitHub user devozerov opened a pull request: https://github.com/apache/ignite/pull/5521 IGNITE-10435 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10435-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5521.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5521 commit 8d2f8bae27befe91af0c7763739d026163a6e3b7 Author: devozerov Date: 2018-11-28T08:40:29Z Removing. commit 0379f945cbd5c095d9cfb84ac01f9a0d97334f4f Author: devozerov Date: 2018-11-28T08:42:54Z Done. ---
[jira] [Created] (IGNITE-10437) GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThreadWithCoalescing is flaky
Ryabov Dmitrii created IGNITE-10437:
---
Summary:
GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThreadWithCoalescing
is flaky
Key: IGNITE-10437
URL: https://issues.apache.org/jira/browse/IGNITE-10437
Project: Ignite
Issue Type: Test
Reporter: Ryabov Dmitrii
Assignee: Ryabov Dmitrii
Fails periodically on
[TeamCity|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2991182438861864832=testDetails_IgniteTests24Java8=%3Cdefault%3E].
{code:java}
junit.framework.AssertionFailedError: No cache overflows detected (a bug or too
few keys or too few delay?)
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThread(GridCacheWriteBehindStoreMultithreadedSelfTest.java:215)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStoreMultithreadedSelfTest.testFlushFromTheSameThreadWithCoalescing(GridCacheWriteBehindStoreMultithreadedSelfTest.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at
org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:150)
at
org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2104)
at
org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2119)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #5520: IGNITE-10376
GitHub user 1vanan opened a pull request: https://github.com/apache/ignite/pull/5520 IGNITE-10376 You can merge this pull request into a Git repository by running: $ git pull https://github.com/1vanan/ignite IGNITE-10376 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5520 commit 0aedd420330cfdae5b35550d938751a9e2070502 Author: Fedotov Date: 2018-11-23T14:15:05Z initial ---
[jira] [Created] (IGNITE-10436) [TC Bot] Add ticket and PR links on report TC Bot page
PetrovMikhail created IGNITE-10436: -- Summary: [TC Bot] Add ticket and PR links on report TC Bot page Key: IGNITE-10436 URL: https://issues.apache.org/jira/browse/IGNITE-10436 Project: Ignite Issue Type: Task Reporter: PetrovMikhail Assignee: PetrovMikhail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10434) MVCC: Transaction asynchronous rollback bug.
Roman Kondakov created IGNITE-10434:
---
Summary: MVCC: Transaction asynchronous rollback bug.
Key: IGNITE-10434
URL: https://issues.apache.org/jira/browse/IGNITE-10434
Project: Ignite
Issue Type: Bug
Components: mvcc
Reporter: Roman Kondakov
There is some bug in mvcc tx asynchronous rollback flow. Sometimes transaction
is not rolled back completely.
Reproducer: {{MvccTxRollbackAsyncTest#testMixedAsyncRollbackTypes}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
