Re: Tx lock partial happens before
Anton, Step-by-step: 1) primary locked on key mention (get/put) at pessimistic/!read-committed tx 2) backups locked on prepare 3) primary unlocked on finish 4) backups unlocked on finish (after the primary) correct? Yes, this corresponds to my understanding of transactions protocol. With minor exception: steps 3 and 4 are inverted in case of one-phase commit. Agree, but seems there is no need to acquire the lock, we have just to wait until entry becomes unlocked. - entry locked means that previous tx's "finish" phase is in progress - entry unlocked means reading value is up-to-date (previous "finish" phase finished) correct? Diving deeper, entry is locked if its GridCacheMapEntry.localCandidates queue is not empty (first item in queue is actually the transaction that owns lock). we have just to wait until entry becomes unlocked. This may work. If consistency checking code has acquired lock on primary, backup can be in two states: - not locked - and new locks won't appear as we are holding lock on primary - still locked by transaction that owned lock on primary just before our checking code - in such case checking code should just wait for lock release Best Regards, Ivan Rakov On 15.07.2019 9:34, Anton Vinogradov wrote: Ivan R. Thanks for joining! Got an idea, but not sure that got a way of a fix. AFAIK (can be wrong, please correct if necessary), at 2PC, locks are acquired on backups during the "prepare" phase and released at "finish" phase after primary fully committed. Step-by-step: 1) primary locked on key mention (get/put) at pessimistic/!read-committed tx 2) backups locked on prepare 3) primary unlocked on finish 4) backups unlocked on finish (after the primary) correct? So, acquiring locks on backups, not at the "prepare" phase, may cause unexpected behavior in case of primary fail or other errors. That's definitely possible to update failover to solve this issue, but it seems to be an overcomplicated way. The main question there, it there any simple way? checking read from backup will just wait for commit if it's in progress. Agree, but seems there is no need to acquire the lock, we have just to wait until entry becomes unlocked. - entry locked means that previous tx's "finish" phase is in progress - entry unlocked means reading value is up-to-date (previous "finish" phase finished) correct? On Mon, Jul 15, 2019 at 8:37 AM Павлухин Иван wrote: Anton, I did not know mechanics locking entries on backups during prepare phase. Thank you for pointing that out! пт, 12 июл. 2019 г. в 22:45, Ivan Rakov : Hi Anton, Each get method now checks the consistency. Check means: 1) tx lock acquired on primary 2) gained data from each owner (primary and backups) 3) data compared Did you consider acquiring locks on backups as well during your check, just like 2PC prepare does? If there's HB between steps 1 (lock primary) and 2 (update primary + lock backup + update backup), you may be sure that there will be no false-positive results and no deadlocks as well. Protocol won't be complicated: checking read from backup will just wait for commit if it's in progress. Best Regards, Ivan Rakov On 12.07.2019 9:47, Anton Vinogradov wrote: Igniters, Let me explain problem in detail. Read Repair at pessimistic tx (locks acquired on primary, full sync, 2pc) able to see consistency violation because backups are not updated yet. This seems to be not a good idea to "fix" code to unlock primary only when backups updated, this definitely will cause a performance drop. Currently, there is no explicit sync feature allows waiting for backups updated during the previous tx. Previous tx just sends GridNearTxFinishResponse to the originating node. Bad ideas how to handle this: - retry some times (still possible to gain false positive) - lock tx entry on backups (will definitely break failover logic) - wait for same entry version on backups during some timeout (will require huge changes at "get" logic and false positive still possible) Is there any simple fix for this issue? Thanks for tips in advance. Ivan, thanks for your interest 4. Very fast and lucky txB writes a value 2 for the key on primary and backup. AFAIK, reordering not possible since backups "prepared" before primary releases lock. So, consistency guaranteed by failover and by "prepare" feature of 2PC. Seems, the problem is NOT with consistency at AI, but with consistency detection implementation (RR) and possible "false positive" results. BTW, checked 1PC case (only one data node at test) and gained no issues. On Fri, Jul 12, 2019 at 9:26 AM Павлухин Иван wrote: Anton, Is such behavior observed for 2PC or for 1PC optimization? Does not it mean that the things can be even worse and an inconsistent write is possible on a backup? E.g. in scenario: 1. txA writes a value 1 for the key on primary. 2. txA unlocks the key on primary. 3. txA freezes before updating backup. 4. Very fast and lucky txB writes a value 2 for the
[jira] [Created] (IGNITE-11985) .NET: CompiledQuery.Compile does not work with string parameters
Pavel Tupitsyn created IGNITE-11985:
---
Summary: .NET: CompiledQuery.Compile does not work with string
parameters
Key: IGNITE-11985
URL: https://issues.apache.org/jira/browse/IGNITE-11985
Project: Ignite
Issue Type: Bug
Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
String parameters always produce an exception, example:
{code}
CompiledQuery.Compile((string empName) => persons.Where(x => x.Value.Name ==
empName))
{code}
Result:
{code}
System.InvalidOperationException: Error compiling query: entire LINQ expression
should be specified within lambda passed to Compile method. Part of the query
can't be outside the Compile method call.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IGNITE-11984) .NET CompiledQuery won't work with strings
Alexandr Shapkin created IGNITE-11984: - Summary: .NET CompiledQuery won't work with strings Key: IGNITE-11984 URL: https://issues.apache.org/jira/browse/IGNITE-11984 Project: Ignite Issue Type: Improvement Components: platforms Affects Versions: 2.7.5 Reporter: Alexandr Shapkin Consider two samples: var cache = GetClientCache(); var persons = cache.AsCacheQueryable(); This works: var qry = CompiledQuery.Compile((int id) => persons.Where(x => x.Value.Id == id)); This won't: var qry = CompiledQuery.Compile((string id) => persons.Where(x => x.Value.Name.Equals(id))); Error compiling query: entire LINQ expression should be specified within lambda passed to Compile method. Reason: GetCompiledQuery method -> var paramValues becomes null Because of that CacheQueryExpressionVisitor -> VisitConstant will not be executed -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IGNITE-11983) TcpDiscoverySpi: Infinite circulation of TcpStatusCheckMessage after node restart
Dmitry Lazurkin created IGNITE-11983:
Summary: TcpDiscoverySpi: Infinite circulation of
TcpStatusCheckMessage after node restart
Key: IGNITE-11983
URL: https://issues.apache.org/jira/browse/IGNITE-11983
Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 2.7
Reporter: Dmitry Lazurkin
Node *ignite-2*/*ignite-3* has inifite logs about
TcpDiscoveryStatusCheckMessage with id
832a931eb61-f507bad3-ea78-40a6-81ce-ab415ab871bb.
h2. Cluster topology
*ignite-1 (f170bfc0-3dec-4b7b-8815-99ca641abc2f)*
{noformat}
2019-07-15 12:13:16,076 [RMI TCP Connection(10)-127.0.0.1] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - TcpDiscoveryNodesRing [
locNode=TcpDiscoveryNode [id=f170bfc0-3dec-4b7b-8815-99ca641abc2f,
addrs=[10.48.44.1], sockAddrs=[ignite-1/10.48.44.1:47500], discPort=47500,
order=2, intOrder=2, lastExchangeTime=1563192795758, loc=true,
ver=2.7.0#20181130-sha1:256ae401, isClient=false],
nodes=[
TcpDiscoveryNode [id=d47598e1-bbfd-4789-ba25-b1af6dc147bf, addrs=[10.48.44.2],
sockAddrs=[ignite-2/10.48.44.2:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1562343474727, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=false],
TcpDiscoveryNode [id=f170bfc0-3dec-4b7b-8815-99ca641abc2f, addrs=[10.48.44.1],
sockAddrs=[ignite-1/10.48.44.1:47500], discPort=47500, order=2, intOrder=2,
lastExchangeTime=1563192795758, loc=true, ver=2.7.0#20181130-sha1:256ae401,
isClient=false],
TcpDiscoveryNode [id=b400aa0a-1d9b-4165-96ae-d75aba2d1bb8,
addrs=[10.40.105.88], sockAddrs=[/10.40.105.88:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1562343561117, loc=false,
ver=2.7.0#20181130-sha1:256ae401, isClient=true],
TcpDiscoveryNode [id=618b2910-071d-4247-86f3-1d40237e7645, addrs=[10.44.151.8],
sockAddrs=[/10.44.151.8:0], discPort=0, order=9, intOrder=7,
lastExchangeTime=1562343652607, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=true],
TcpDiscoveryNode [id=2b033aed-132d-4341-b1cf-6f45a21619b7, addrs=[10.37.26.56],
sockAddrs=[/10.37.26.56:0], discPort=0, order=14, intOrder=10,
lastExchangeTime=1562851741145, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=true],
TcpDiscoveryNode [id=6a182cef-25dc-4ee6-8307-a62ec164f466, addrs=[10.48.44.3],
sockAddrs=[ignite-3/10.48.44.3:47500], discPort=47500, order=18, intOrder=12,
lastExchangeTime=1562853582125, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=false]], topVer=18, nodeOrder=12, maxInternalOrder=12,
minNodeVer=2.7.0#20181130-sha1:256ae401]
{noformat}
*ignite-2 (d47598e1-bbfd-4789-ba25-b1af6dc147bf, coordinator)*
{noformat}
2019-07-15 12:11:29,679 [RMI TCP Connection(88)-127.0.0.1] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - TcpDiscoveryNodesRing [
locNode=TcpDiscoveryNode [id=d47598e1-bbfd-4789-ba25-b1af6dc147bf,
addrs=[10.48.44.2], sockAddrs=[ignite-2/10.48.44.2:47500], discPort=47500,
order=1, intOrder=1, lastExchangeTime=1563192689675, loc=true,
ver=2.7.0#20181130-sha1:256ae401, isClient=false],
nodes=[
TcpDiscoveryNode [id=d47598e1-bbfd-4789-ba25-b1af6dc147bf, addrs=[10.48.44.2],
sockAddrs=[ignite-2/10.48.44.2:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1563192689675, loc=true, ver=2.7.0#20181130-sha1:256ae401,
isClient=false],
TcpDiscoveryNode [id=f170bfc0-3dec-4b7b-8815-99ca641abc2f, addrs=[10.48.44.1],
sockAddrs=[ignite-1/10.48.44.1:47500], discPort=47500, order=2, intOrder=2,
lastExchangeTime=1562343474657, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=false],
TcpDiscoveryNode [id=b400aa0a-1d9b-4165-96ae-d75aba2d1bb8,
addrs=[10.40.105.88], sockAddrs=[/10.40.105.88:0], discPort=0, order=7,
intOrder=6, lastExchangeTime=1562343561102, loc=false,
ver=2.7.0#20181130-sha1:256ae401, isClient=true],
TcpDiscoveryNode [id=618b2910-071d-4247-86f3-1d40237e7645, addrs=[10.44.151.8],
sockAddrs=[/10.44.151.8:0], discPort=0, order=9, intOrder=7,
lastExchangeTime=1562343652597, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=true],
TcpDiscoveryNode [id=2b033aed-132d-4341-b1cf-6f45a21619b7, addrs=[10.37.26.56],
sockAddrs=[/10.37.26.56:0], discPort=0, order=14, intOrder=10,
lastExchangeTime=1562851741133, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=true],
TcpDiscoveryNode [id=6a182cef-25dc-4ee6-8307-a62ec164f466, addrs=[10.48.44.3],
sockAddrs=[ignite-3/10.48.44.3:47500], discPort=47500, order=18, intOrder=12,
lastExchangeTime=1562853582098, loc=false, ver=2.7.0#20181130-sha1:256ae401,
isClient=false]], topVer=18, nodeOrder=12, maxInternalOrder=12,
minNodeVer=2.7.0#20181130-sha1:256ae401]
{noformat}
*ignite-3 (6a182cef-25dc-4ee6-8307-a62ec164f466, id before restart -
f507bad3-ea78-40a6-81ce-ab415ab871bb)*
{noformat}
2019-07-15 12:06:33,828 [RMI TCP Connection(39)-127.0.0.1] INFO
o.a.i.s.d.tcp.TcpDiscoverySpi - TcpDiscoveryNodesRing [
locNode=TcpDiscoveryNode [id=6a182cef-25dc-4ee6-8307-a62ec164f466,
addrs=[10.48.44.3],
Re: Read Repair (ex. Consistency Check) - review request #2
Ok, thank you
пн, 15 июл. 2019 г., 16:46 Nikolay Izhikov :
> I did the review.
>
> пн, 15 июля 2019 г., 16:15 Dmitriy Pavlov :
>
> > Igniters, who did a review of
> > https://issues.apache.org/jira/browse/IGNITE-10663 before the merge?
> I've
> > checked both PR https://github.com/apache/ignite/pull/5656 and Issue,
> > and dev.list thread and didn't find any LGTM.
> >
> > Anton, since you've rejected lazy consensus in our process, we have RTC
> in
> > that (core) module. So I'd like to know if the fix was covered by the
> > review.
> >
> > Because you're a committer, a reviewer can be non-committer. So, who was
> a
> > reviewer? Or was process ignored?
> >
> > пн, 15 июл. 2019 г. в 15:37, Вячеслав Коптилин >:
> >
> > > Hello Anton,
> > >
> > > > I'd like to propose you to provide fixes as a PR since you have a
> > vision
> > > of how it should be made. I'll review them and merge shortly.
> > > Could you please take a look at PR:
> > > https://github.com/apache/ignite/pull/6689
> > >
> > > > Since your comments mostly about Javadoc (does this mean that my
> > solution
> > > is so great that you ask me only to fix Javadocs :) ?),
> > > In my humble opinion, I would consider this feature as experimental one
> > (It
> > > does not seem production-ready).
> > > Let me clarify this with the following simple example:
> > >
> > > try {
> > > atomicCache.withReadRepair().getAll(keys);
> > > }
> > > catch (CacheException e) {
> > > // What should be done here from the end-user point of view?
> > > }
> > >
> > > 1. Should I consider that my cluster is broken? There is no answer! The
> > > false-positive result is possible.
> > > 2. What should be done here in order to check/resolve the issue?
> > Perhaps, I
> > > should restart a node (which one?), restart the whole cluster, put a
> new
> > > value...
> > > 3. IgniteConsistencyViolationException is absolutely useless. It does
> not
> > > provide any information about the issue and possible way to fix it.
> > >
> > > It seems that transactional caches are covered much better.
> > >
> > > > Mostly agree with you, but
> > > > - MVCC is not production ready,
> > > > - not sure near support really required,
> > > > - metrics are better for monitoring, but the Event is enough for my
> > wish
> > > to
> > > > cover AI with consistency check,
> > > > - do we really need Platforms and Thin Client support?
> > > Well, near caches are widely used and fully transactional, so I think
> it
> > > makes sense to support the feature for near caches too.
> > > .Net is already aware of 'ReadRepair'. It seems to me, that it can be
> > > easily supported for C++. I don't see a reason why it should not be
> done
> > :)
> > >
> > > > Do you mean per partition check and recovery? That's a good idea,
> but I
> > > found it's not easy to imagine API to for such tool.
> > > Yep, perhaps it can be done on the idle cluster via `idle-verify`
> command
> > > with additional flag. Agreed, that this approach is not the best one
> > > definitely.
> > >
> > > Thanks,
> > > S.
> > >
> > > чт, 11 июл. 2019 г. в 09:53, Anton Vinogradov :
> > >
> > > > Slava,
> > > >
> > > > Thanks for your review first!
> > > >
> > > > >> Anyway, I left some comments in your pull-request at github.
> Please
> > > take
> > > > a
> > > > >> look. The most annoying thing is poorly documented code :(
> > > > Since your comments mostly about Javadoc (does this mean that my
> > solution
> > > > is so great that you ask me only to fix Javadocs :) ?),
> > > > I'd like to propose you to provide fixes as a PR since you have a
> > vision
> > > of
> > > > how it should be made. I'll review them and merge shortly.
> > > >
> > > > >> By the way, is it required to add test related to fail-over
> > scenarios?
> > > > The best check is to use RR at real code.
> > > > For example, I'm injecting RR now to the test with concurrent
> > > modifications
> > > > and restarts [1].
> > > >
> > > > >> I just checked, the IEP page and I still cannot find Jira tickets
> > that
> > > > >> should cover existing limitations/improvements.
> > > > >> I would suggest creating the following tasks, at least:
> > > > Mostly agree with you, but
> > > > - MVCC is not production ready,
> > > > - not sure near support really required,
> > > > - metrics are better for monitoring, but the Event is enough for my
> > wish
> > > to
> > > > cover AI with consistency check,
> > > > - do we really need Platforms and Thin Client support?
> > > > Also, we should not produce stillborn issue.
> > > > All limitations listed at proxy creation method and they definitely
> are
> > > not
> > > > showstoppers and may be fixed later if someone interested.
> > > > Сoming back to AI 3.0 discussion, we have A LOT of features and it's
> > > almost
> > > > impossible (require much more time that feature's cost) to support
> them
> > > > all.
> > > > I will be pretty happy in case someone will do this and provide help
> if
> > > > necessary!
> > >
Re: Read Repair (ex. Consistency Check) - review request #2
I did the review.
пн, 15 июля 2019 г., 16:15 Dmitriy Pavlov :
> Igniters, who did a review of
> https://issues.apache.org/jira/browse/IGNITE-10663 before the merge? I've
> checked both PR https://github.com/apache/ignite/pull/5656 and Issue,
> and dev.list thread and didn't find any LGTM.
>
> Anton, since you've rejected lazy consensus in our process, we have RTC in
> that (core) module. So I'd like to know if the fix was covered by the
> review.
>
> Because you're a committer, a reviewer can be non-committer. So, who was a
> reviewer? Or was process ignored?
>
> пн, 15 июл. 2019 г. в 15:37, Вячеслав Коптилин :
>
> > Hello Anton,
> >
> > > I'd like to propose you to provide fixes as a PR since you have a
> vision
> > of how it should be made. I'll review them and merge shortly.
> > Could you please take a look at PR:
> > https://github.com/apache/ignite/pull/6689
> >
> > > Since your comments mostly about Javadoc (does this mean that my
> solution
> > is so great that you ask me only to fix Javadocs :) ?),
> > In my humble opinion, I would consider this feature as experimental one
> (It
> > does not seem production-ready).
> > Let me clarify this with the following simple example:
> >
> > try {
> > atomicCache.withReadRepair().getAll(keys);
> > }
> > catch (CacheException e) {
> > // What should be done here from the end-user point of view?
> > }
> >
> > 1. Should I consider that my cluster is broken? There is no answer! The
> > false-positive result is possible.
> > 2. What should be done here in order to check/resolve the issue?
> Perhaps, I
> > should restart a node (which one?), restart the whole cluster, put a new
> > value...
> > 3. IgniteConsistencyViolationException is absolutely useless. It does not
> > provide any information about the issue and possible way to fix it.
> >
> > It seems that transactional caches are covered much better.
> >
> > > Mostly agree with you, but
> > > - MVCC is not production ready,
> > > - not sure near support really required,
> > > - metrics are better for monitoring, but the Event is enough for my
> wish
> > to
> > > cover AI with consistency check,
> > > - do we really need Platforms and Thin Client support?
> > Well, near caches are widely used and fully transactional, so I think it
> > makes sense to support the feature for near caches too.
> > .Net is already aware of 'ReadRepair'. It seems to me, that it can be
> > easily supported for C++. I don't see a reason why it should not be done
> :)
> >
> > > Do you mean per partition check and recovery? That's a good idea, but I
> > found it's not easy to imagine API to for such tool.
> > Yep, perhaps it can be done on the idle cluster via `idle-verify` command
> > with additional flag. Agreed, that this approach is not the best one
> > definitely.
> >
> > Thanks,
> > S.
> >
> > чт, 11 июл. 2019 г. в 09:53, Anton Vinogradov :
> >
> > > Slava,
> > >
> > > Thanks for your review first!
> > >
> > > >> Anyway, I left some comments in your pull-request at github. Please
> > take
> > > a
> > > >> look. The most annoying thing is poorly documented code :(
> > > Since your comments mostly about Javadoc (does this mean that my
> solution
> > > is so great that you ask me only to fix Javadocs :) ?),
> > > I'd like to propose you to provide fixes as a PR since you have a
> vision
> > of
> > > how it should be made. I'll review them and merge shortly.
> > >
> > > >> By the way, is it required to add test related to fail-over
> scenarios?
> > > The best check is to use RR at real code.
> > > For example, I'm injecting RR now to the test with concurrent
> > modifications
> > > and restarts [1].
> > >
> > > >> I just checked, the IEP page and I still cannot find Jira tickets
> that
> > > >> should cover existing limitations/improvements.
> > > >> I would suggest creating the following tasks, at least:
> > > Mostly agree with you, but
> > > - MVCC is not production ready,
> > > - not sure near support really required,
> > > - metrics are better for monitoring, but the Event is enough for my
> wish
> > to
> > > cover AI with consistency check,
> > > - do we really need Platforms and Thin Client support?
> > > Also, we should not produce stillborn issue.
> > > All limitations listed at proxy creation method and they definitely are
> > not
> > > showstoppers and may be fixed later if someone interested.
> > > Сoming back to AI 3.0 discussion, we have A LOT of features and it's
> > almost
> > > impossible (require much more time that feature's cost) to support them
> > > all.
> > > I will be pretty happy in case someone will do this and provide help if
> > > necessary!
> > >
> > > >> Perhaps, it would be a good idea to think about the recovery too
> > > Do you mean per partition check and recovery?
> > > That's a good idea, but I found it's not easy to imagine API to for
> such
> > > tool.
> > > In case you ready to assist with proper API/design this will definitely
> > > help.
> > >
> > > [1]
[jira] [Created] (IGNITE-11982) Fix bugs of pds
Anton Kalashnikov created IGNITE-11982: -- Summary: Fix bugs of pds Key: IGNITE-11982 URL: https://issues.apache.org/jira/browse/IGNITE-11982 Project: Ignite Issue Type: Bug Reporter: Anton Kalashnikov Assignee: Anton Kalashnikov fixed pds crash: * Fail during logical recovery * JVM crash in all compatibility LFS tests * WAL segments serialization problem * Unable to read last WAL record after crash during checkpoint * Node failed on detecting storage block size if page compression enabled on many caches * Can not change baseline for in-memory cluster * SqlFieldsQuery DELETE FROM causes JVM crash * Fixed IgniteCheckedException: Compound exception for CountDownFuture. fixed tests: * WalCompactionAndPageCompressionTest * IgnitePdsRestartAfterFailedToWriteMetaPageTest.test * GridPointInTimeRecoveryRebalanceTest.testRecoveryNotFailsIfWalSomewhereEnab * IgniteClusterActivateDeactivateTest.testDeactivateSimple_5_Servers_5_Clients_Fro * IgniteCacheReplicatedQuerySelfTest.testNodeLeft * .NET tests optimization: * Replace TcpDiscoveryNode to nodeId in TcpDiscoveryMessages * Failures to deserialize discovery data should be handled by a failure handler * Optimize GridToStringBuilder -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: Read Repair (ex. Consistency Check) - review request #2
Igniters, who did a review of
https://issues.apache.org/jira/browse/IGNITE-10663 before the merge? I've
checked both PR https://github.com/apache/ignite/pull/5656 and Issue,
and dev.list thread and didn't find any LGTM.
Anton, since you've rejected lazy consensus in our process, we have RTC in
that (core) module. So I'd like to know if the fix was covered by the
review.
Because you're a committer, a reviewer can be non-committer. So, who was a
reviewer? Or was process ignored?
пн, 15 июл. 2019 г. в 15:37, Вячеслав Коптилин :
> Hello Anton,
>
> > I'd like to propose you to provide fixes as a PR since you have a vision
> of how it should be made. I'll review them and merge shortly.
> Could you please take a look at PR:
> https://github.com/apache/ignite/pull/6689
>
> > Since your comments mostly about Javadoc (does this mean that my solution
> is so great that you ask me only to fix Javadocs :) ?),
> In my humble opinion, I would consider this feature as experimental one (It
> does not seem production-ready).
> Let me clarify this with the following simple example:
>
> try {
> atomicCache.withReadRepair().getAll(keys);
> }
> catch (CacheException e) {
> // What should be done here from the end-user point of view?
> }
>
> 1. Should I consider that my cluster is broken? There is no answer! The
> false-positive result is possible.
> 2. What should be done here in order to check/resolve the issue? Perhaps, I
> should restart a node (which one?), restart the whole cluster, put a new
> value...
> 3. IgniteConsistencyViolationException is absolutely useless. It does not
> provide any information about the issue and possible way to fix it.
>
> It seems that transactional caches are covered much better.
>
> > Mostly agree with you, but
> > - MVCC is not production ready,
> > - not sure near support really required,
> > - metrics are better for monitoring, but the Event is enough for my wish
> to
> > cover AI with consistency check,
> > - do we really need Platforms and Thin Client support?
> Well, near caches are widely used and fully transactional, so I think it
> makes sense to support the feature for near caches too.
> .Net is already aware of 'ReadRepair'. It seems to me, that it can be
> easily supported for C++. I don't see a reason why it should not be done :)
>
> > Do you mean per partition check and recovery? That's a good idea, but I
> found it's not easy to imagine API to for such tool.
> Yep, perhaps it can be done on the idle cluster via `idle-verify` command
> with additional flag. Agreed, that this approach is not the best one
> definitely.
>
> Thanks,
> S.
>
> чт, 11 июл. 2019 г. в 09:53, Anton Vinogradov :
>
> > Slava,
> >
> > Thanks for your review first!
> >
> > >> Anyway, I left some comments in your pull-request at github. Please
> take
> > a
> > >> look. The most annoying thing is poorly documented code :(
> > Since your comments mostly about Javadoc (does this mean that my solution
> > is so great that you ask me only to fix Javadocs :) ?),
> > I'd like to propose you to provide fixes as a PR since you have a vision
> of
> > how it should be made. I'll review them and merge shortly.
> >
> > >> By the way, is it required to add test related to fail-over scenarios?
> > The best check is to use RR at real code.
> > For example, I'm injecting RR now to the test with concurrent
> modifications
> > and restarts [1].
> >
> > >> I just checked, the IEP page and I still cannot find Jira tickets that
> > >> should cover existing limitations/improvements.
> > >> I would suggest creating the following tasks, at least:
> > Mostly agree with you, but
> > - MVCC is not production ready,
> > - not sure near support really required,
> > - metrics are better for monitoring, but the Event is enough for my wish
> to
> > cover AI with consistency check,
> > - do we really need Platforms and Thin Client support?
> > Also, we should not produce stillborn issue.
> > All limitations listed at proxy creation method and they definitely are
> not
> > showstoppers and may be fixed later if someone interested.
> > Сoming back to AI 3.0 discussion, we have A LOT of features and it's
> almost
> > impossible (require much more time that feature's cost) to support them
> > all.
> > I will be pretty happy in case someone will do this and provide help if
> > necessary!
> >
> > >> Perhaps, it would be a good idea to think about the recovery too
> > Do you mean per partition check and recovery?
> > That's a good idea, but I found it's not easy to imagine API to for such
> > tool.
> > In case you ready to assist with proper API/design this will definitely
> > help.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11973
> >
> > On Wed, Jul 10, 2019 at 3:43 PM Вячеслав Коптилин <
> > slava.kopti...@gmail.com>
> > wrote:
> >
> > > Perhaps, it would be a good idea to think about the recovery tool/
> > > control-utility command that will allow achieving the same goal.
> > > If I am not
Re: Read Repair (ex. Consistency Check) - review request #2
Hello Anton,
> I'd like to propose you to provide fixes as a PR since you have a vision
of how it should be made. I'll review them and merge shortly.
Could you please take a look at PR:
https://github.com/apache/ignite/pull/6689
> Since your comments mostly about Javadoc (does this mean that my solution
is so great that you ask me only to fix Javadocs :) ?),
In my humble opinion, I would consider this feature as experimental one (It
does not seem production-ready).
Let me clarify this with the following simple example:
try {
atomicCache.withReadRepair().getAll(keys);
}
catch (CacheException e) {
// What should be done here from the end-user point of view?
}
1. Should I consider that my cluster is broken? There is no answer! The
false-positive result is possible.
2. What should be done here in order to check/resolve the issue? Perhaps, I
should restart a node (which one?), restart the whole cluster, put a new
value...
3. IgniteConsistencyViolationException is absolutely useless. It does not
provide any information about the issue and possible way to fix it.
It seems that transactional caches are covered much better.
> Mostly agree with you, but
> - MVCC is not production ready,
> - not sure near support really required,
> - metrics are better for monitoring, but the Event is enough for my wish
to
> cover AI with consistency check,
> - do we really need Platforms and Thin Client support?
Well, near caches are widely used and fully transactional, so I think it
makes sense to support the feature for near caches too.
.Net is already aware of 'ReadRepair'. It seems to me, that it can be
easily supported for C++. I don't see a reason why it should not be done :)
> Do you mean per partition check and recovery? That's a good idea, but I
found it's not easy to imagine API to for such tool.
Yep, perhaps it can be done on the idle cluster via `idle-verify` command
with additional flag. Agreed, that this approach is not the best one
definitely.
Thanks,
S.
чт, 11 июл. 2019 г. в 09:53, Anton Vinogradov :
> Slava,
>
> Thanks for your review first!
>
> >> Anyway, I left some comments in your pull-request at github. Please take
> a
> >> look. The most annoying thing is poorly documented code :(
> Since your comments mostly about Javadoc (does this mean that my solution
> is so great that you ask me only to fix Javadocs :) ?),
> I'd like to propose you to provide fixes as a PR since you have a vision of
> how it should be made. I'll review them and merge shortly.
>
> >> By the way, is it required to add test related to fail-over scenarios?
> The best check is to use RR at real code.
> For example, I'm injecting RR now to the test with concurrent modifications
> and restarts [1].
>
> >> I just checked, the IEP page and I still cannot find Jira tickets that
> >> should cover existing limitations/improvements.
> >> I would suggest creating the following tasks, at least:
> Mostly agree with you, but
> - MVCC is not production ready,
> - not sure near support really required,
> - metrics are better for monitoring, but the Event is enough for my wish to
> cover AI with consistency check,
> - do we really need Platforms and Thin Client support?
> Also, we should not produce stillborn issue.
> All limitations listed at proxy creation method and they definitely are not
> showstoppers and may be fixed later if someone interested.
> Сoming back to AI 3.0 discussion, we have A LOT of features and it's almost
> impossible (require much more time that feature's cost) to support them
> all.
> I will be pretty happy in case someone will do this and provide help if
> necessary!
>
> >> Perhaps, it would be a good idea to think about the recovery too
> Do you mean per partition check and recovery?
> That's a good idea, but I found it's not easy to imagine API to for such
> tool.
> In case you ready to assist with proper API/design this will definitely
> help.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-11973
>
> On Wed, Jul 10, 2019 at 3:43 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> wrote:
>
> > Perhaps, it would be a good idea to think about the recovery tool/
> > control-utility command that will allow achieving the same goal.
> > If I am not mistaken it was already proposed in the email thread.
> >
> > Thanks,
> > S.
> >
> > ср, 10 июл. 2019 г. в 15:33, Вячеслав Коптилин >:
> >
> > > Hi Anton,
> > >
> > > Well, the ReadRepair feature is finally merged and that is good news :)
> > >
> > > Unfortunately, I cannot find a consensus about the whole functionality
> in
> > > any of these topics:
> > > -
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Consistency-check-and-fix-review-request-td41629.html
> > > -
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Read-Repair-ex-Consistency-Check-review-request-2-td42421.html
> > > Also, there are no comments/discussion in JIRA. That makes me sad :(
> > > especially when a feature is huge, not obvious
[jira] [Created] (IGNITE-11981) [IEP-35] Create MU shortcut instead of static import of MetricUtils
Nikolay Izhikov created IGNITE-11981:
Summary: [IEP-35] Create MU shortcut instead of static import of
MetricUtils
Key: IGNITE-11981
URL: https://issues.apache.org/jira/browse/IGNITE-11981
Project: Ignite
Issue Type: Improvement
Reporter: Nikolay Izhikov
More Ignite-way of coding is the usage of short cut classes.
We should use MU instead of static import of {{MetricUtils}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IGNITE-11980) Repairs count metric (ReadRepair)
Anton Vinogradov created IGNITE-11980: - Summary: Repairs count metric (ReadRepair) Key: IGNITE-11980 URL: https://issues.apache.org/jira/browse/IGNITE-11980 Project: Ignite Issue Type: Task Reporter: Anton Vinogradov Ignite should provide metric of how many repairs happened during ReadRepar feature usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: Tx lock partial happens before
Ivan R. Thanks for joining! Got an idea, but not sure that got a way of a fix. AFAIK (can be wrong, please correct if necessary), at 2PC, locks are acquired on backups during the "prepare" phase and released at "finish" phase after primary fully committed. Step-by-step: 1) primary locked on key mention (get/put) at pessimistic/!read-committed tx 2) backups locked on prepare 3) primary unlocked on finish 4) backups unlocked on finish (after the primary) correct? So, acquiring locks on backups, not at the "prepare" phase, may cause unexpected behavior in case of primary fail or other errors. That's definitely possible to update failover to solve this issue, but it seems to be an overcomplicated way. The main question there, it there any simple way? >> checking read from backup will just wait for commit if it's in progress. Agree, but seems there is no need to acquire the lock, we have just to wait until entry becomes unlocked. - entry locked means that previous tx's "finish" phase is in progress - entry unlocked means reading value is up-to-date (previous "finish" phase finished) correct? On Mon, Jul 15, 2019 at 8:37 AM Павлухин Иван wrote: > Anton, > > I did not know mechanics locking entries on backups during prepare > phase. Thank you for pointing that out! > > пт, 12 июл. 2019 г. в 22:45, Ivan Rakov : > > > > Hi Anton, > > > > > Each get method now checks the consistency. > > > Check means: > > > 1) tx lock acquired on primary > > > 2) gained data from each owner (primary and backups) > > > 3) data compared > > Did you consider acquiring locks on backups as well during your check, > > just like 2PC prepare does? > > If there's HB between steps 1 (lock primary) and 2 (update primary + > > lock backup + update backup), you may be sure that there will be no > > false-positive results and no deadlocks as well. Protocol won't be > > complicated: checking read from backup will just wait for commit if it's > > in progress. > > > > Best Regards, > > Ivan Rakov > > > > On 12.07.2019 9:47, Anton Vinogradov wrote: > > > Igniters, > > > > > > Let me explain problem in detail. > > > Read Repair at pessimistic tx (locks acquired on primary, full sync, > 2pc) > > > able to see consistency violation because backups are not updated yet. > > > This seems to be not a good idea to "fix" code to unlock primary only > when > > > backups updated, this definitely will cause a performance drop. > > > Currently, there is no explicit sync feature allows waiting for backups > > > updated during the previous tx. > > > Previous tx just sends GridNearTxFinishResponse to the originating > node. > > > > > > Bad ideas how to handle this: > > > - retry some times (still possible to gain false positive) > > > - lock tx entry on backups (will definitely break failover logic) > > > - wait for same entry version on backups during some timeout (will > require > > > huge changes at "get" logic and false positive still possible) > > > > > > Is there any simple fix for this issue? > > > Thanks for tips in advance. > > > > > > Ivan, > > > thanks for your interest > > > > > >>> 4. Very fast and lucky txB writes a value 2 for the key on primary > and > > > backup. > > > AFAIK, reordering not possible since backups "prepared" before primary > > > releases lock. > > > So, consistency guaranteed by failover and by "prepare" feature of 2PC. > > > Seems, the problem is NOT with consistency at AI, but with consistency > > > detection implementation (RR) and possible "false positive" results. > > > BTW, checked 1PC case (only one data node at test) and gained no > issues. > > > > > > On Fri, Jul 12, 2019 at 9:26 AM Павлухин Иван > wrote: > > > > > >> Anton, > > >> > > >> Is such behavior observed for 2PC or for 1PC optimization? Does not it > > >> mean that the things can be even worse and an inconsistent write is > > >> possible on a backup? E.g. in scenario: > > >> 1. txA writes a value 1 for the key on primary. > > >> 2. txA unlocks the key on primary. > > >> 3. txA freezes before updating backup. > > >> 4. Very fast and lucky txB writes a value 2 for the key on primary and > > >> backup. > > >> 5. txB wakes up and writes 1 for the key. > > >> 6. As result there is 2 on primary and 1 on backup. > > >> > > >> Naively it seems that locks should be released after all replicas are > > >> updated. > > >> > > >> ср, 10 июл. 2019 г. в 16:36, Anton Vinogradov : > > >>> Folks, > > >>> > > >>> Investigating now unexpected repairs [1] in case of ReadRepair usage > at > > >>> testAccountTxNodeRestart. > > >>> Updated [2] the test to check is there any repairs happen. > > >>> Test's name now is "testAccountTxNodeRestartWithReadRepair". > > >>> > > >>> Each get method now checks the consistency. > > >>> Check means: > > >>> 1) tx lock acquired on primary > > >>> 2) gained data from each owner (primary and backups) > > >>> 3) data compared > > >>> > > >>> Sometime, backup may have obsolete value during such check. > > >>> > > >>> Seems, this happen because
